DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 3/27/2025 and 7/23/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5,7-11 and 13-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fisher(US-20130310951-A1) in view of Evans(US-20210110598-A1).
Regarding claim1, Fisher discloses An experience platform system comprising: a physical world element(para.[0045] “FIG. 1 shows an exemplary embodiment of the automation and motion control system of the present application. The automation and control system 100 can include a real time network 110 interconnecting operator consoles 115, remote stations 120, safety systems 125, machinery 130, input/output devices 135 and external systems 140. In one exemplary embodiment, safety systems 125 can include emergency stop (e-stop) systems; machinery 130 can include lifts, chain hoists, winches, elevators, carousels, turntables, hydraulic systems, pneumatic systems, multi-axis systems, linear motion systems (e.g., deck tracks and line sets),” Fisher details the claimed physical world element, describing the tangible physical devices involved in the taught control system.); However, Fisher does not fully disclose a virtual world element; and an experience controller integrating the physical world element and the virtual world element; wherein the integrating includes real-time control of one or both of the physical world element and the virtual world element.
The combination of Fisher and Evans does disclose a virtual world element(Evans at para.[0014] “For example, the motion determination module can track the motion of objects or users in the MR environment and can generate virtual articles, such as virtual effects or lights, which can be displayed in an MR display, where the virtual objects move or otherwise change with the motion of the objects or users. As another example, as virtual object moves or changes brightness, the physical articles are actuated correspondingly to generate a combined effect for a user, where the overall effect will enhance the impact and realism of the virtual object.” Evans describe virtual articles, aligning with this claim element.); and an experience controller integrating the physical world element and the virtual world element(Evans teaches at para[0024] “The server 104 or computing element controls various aspects and components of the system 100, e.g., the server 104 can coordinate operation of modules to generate various physical 120 and/or virtual 118 articles”); wherein the integrating includes real-time control of one or both of the physical world element and the virtual world element(para[0041] “The mapping is used to generate a model that corresponds to the physical environment and is used to track and actuate virtual and physical articles in coordination with one another. The environment model includes information corresponding to both the physical environment and the corresponding virtual environment. Objects and articles within both environments can then be tracked and coordinated within the combined environment to generate corresponding effects For example, the server 104 can use the mapping as well as sensors and/or monitors to capture detailed state information about the mixed reality environment 102. In particular, the state of physical articles 120 like physical light sources and virtual articles 118 like virtual light sources effects or animations can be monitored by the server 104 and used to generate or update the environment model. The server 104 updates the actuation of physical article 120 or virtual article 1180 based on state changes;” Evans further teaches at para[0044] “Based on the determined state information for the physical article 120 and/or virtual article 118, as well as the model environment, the method 200 may proceed substantially simultaneously to operations 206 and 208 and the system 100 renders or generates physical articles 120 and virtual articles 118, respectively. In other words, operations 206 and 208 may be executed in parallel … In some instances, the environment may also accommodate real time tracking and updating of physical/virtual articles based on un-programmed or planned content. As an example, using the model, light interactions between physical 120 and virtual 118 articles can be extrapolated, e.g., as a virtual article is actuated, the physical articles are actuated based on an estimated impact of the virtual article in the physical environment.” Here, Evans teaches “real time” physical and virtual articles and their control.) It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Evans into the teachings of Fisher to produce coordinating physical and virtual effects. Fisher provides real-time distributed control infrastructure, while Evans teaches using a combines physical and virtual environment model to control one domain in response to the other.
Regarding claim 2, the combination of Fisher and Evans disclose all the elements of claim 1 as discussed above. Evans also discloses wherein the integrating includes real-time control of the physical world element includes a response to a condition of the virtual world element(para.[0044] “Based on the determined state information for the physical article 120 and/or virtual article 118, as well as the model environment, the method 200 may proceed substantially simultaneously to operations 206 and 208 and the system 100 renders or generates physical articles 120 and virtual articles 118, respectively. In other words, operations 206 and 208 may be executed in parallel. The actuation and selection of particular physical and virtual articles depends on various inputs, frames, timing, or the like, with respect to either the virtual or physical environment. For example, if a new frame in the virtual environment is selected to be rendered at the next moment in time, the corresponding physical article corresponding to the new frame (e.g., light effects based on objects newly presented in the new frame) is selected to be rendered next well.” Evans gives specific examples where a new virtual fram causes the corresponding physical article to be selected and actuated next. That supples the real-time aspect and the causal relationship required by the claim: a condition or change in the virtual world element drives control of the physical world element.) The rationale from claim 1 is incorporated herein.
Regarding claim 3, the combination of Fisher and Evans disclose all the elements of claim 1 as discussed above. Fisher also discloses wherein physical world elements include one or both of a building management system (BMS)(para.[0046]“ In one exemplary embodiment, safety systems 125 can include emergency stop (e-stop) systems; machinery 130 can include lifts… ControlNet systems, DeviceNet systems, RS 232 systems, RS 45 systems, CAN bus (Controller Area Network bus) systems, Maya systems, Lightwave systems, Catalyst systems, 3ds Max or 3D Studio Max systems, and/or a custom designed system.” Here, Fisher explicitly discloses a physical management system that interacts with the virtual content as discussed above. Since the claim recites one or both only one needs to be disclosed for the claim to be rejected.) and a content management system (CMS). The rationale from claim 1 is incorporated herein.
Regarding claim 4, the combination of Fisher and Evans disclose all the elements of claim1 as discussed above. Fisher also discloses wherein physical world elements may be selected from the group consisting of motors, drivers, cameras, computer vision (CV),sensors, lighting elements, sound/acoustic elements, pyrotechnic elements, video screens, point of sale systems, mobile devices, wearables and combinations thereof(Fisher discloses at para.[0053] “The machinery can be operated by hydraulic, electric, or pneumatic systems. The control system can be used with a variety of different axis devices or nodes that correspond to the controllers for the end machines that make theatrical objects move. Examples of axis machinery can include engines, motors (AC/DC), servos, hydraulic movers, and pneumatic movers.” Fisher teaches items explicitly stated in the claim limitation as well as in a group aligning with the claim language.)
Regarding claim 5, the combination of Fisher and Evans disclose all the elements of claim 1 as discussed above. Evans also discloses wherein virtual world elements may be selected from the group consisting of user profiles, virtual spaces, virtual models, simulators, physics engines, game engines, avatars, applications, big data, artificial intelligence, and combinations thereof. (para.[0041] “The mapping is used to generate a model that corresponds to the physical environment and is used to track and actuate virtual and physical articles in coordination with one another. The environment model includes information corresponding to both the physical environment and the corresponding virtual environment.” Further at para.[0048] “Virtual articles 118 can be virtual lights, animations, objects, or other effects that generally would not occur in the physical world. Some examples include: light or energy beams, fluids that appear to defy gravity, animated characters (either fictitious or based on real people), animals, or even simulations like flying, running, swimming, or space travel.” Evans explicitly discloses elements and some combinations of the claim limitations (virtual spaces, models and avatars).) The rationale from claim 1 is incorporated herein.
Regarding claim 13, Fisher discloses An experience platform system for providing control of one or both of the physical world element and the virtual world element, the system comprising:a plurality of nodes and an operator console node in communication with each other over a real time network(para.[0045] “FIG. 1 shows an exemplary embodiment of the automation and motion control system of the present application. The automation and control system 100 can include a real time network 110 interconnecting operator consoles 115, remote stations 120, safety systems 125, machinery 130, input/output devices 135 and external systems 140.” Para.[0047] “As shown in FIG. 2, nodes 210 and operator console nodes 215 are interconnected with each other. Thus, any node 210, 215 can communicate, i.e., send and receive data and/or instructions, with any other node 210, 215 in the control system 100.” Fisher provides a plurality of nodes, an operator console node and their communication over a real-time network.); one or both of a node of the plurality of nodes or the operator console node corresponds to the physical world element(para.[0045] “Each node 210 can be a specific device (or group of devices) from remote stations 120, safety systems 125, machinery 130, input/output devices 135 and external systems 140. An operator console node 215 can be a specific device from operator consoles 115 and can enable an operator to interact with the control system 100, i.e., to send data and instructions to the control system 100 and to receive data and information from the control system 100. ” Fisher teaches a node corresponding to a physical-world element, such as a motor, winch, lighting device or other piece of physical machinery.);
However, Fisher along does not disclose one or both of a node of the plurality of nodes or the operator console node corresponds to virtual world element; each node of the plurality of nodes and the operator console node comprising a microprocessor and a memory device, the operator console node further comprising an integrator for integrating the physical world element and the virtual world element
The combination of Fisher and Evans does disclose one or both of a node of the plurality of nodes or the operator console node corresponds to virtual world element(“Fisher at para.[0052] “Each node in column 402 can include a computing device such as a process controller, a thin client, a palm top computer, single board computer (SBC), programmable logic controller (PLC), field programmable gate array (FPGA) device or a microprocessor, or the node can be a software device like a “player” or a “virtual machine” which is not a tangible piece of machinery capable of being held in one's hand but is instead a software construct that is considered by the control system to be a device. The software device can supply commands and receive back a status, yet doesn't exist as a physical device.” Evan then discloses at para[0041] “The mapping is used to generate a model that corresponds to the physical environment and is used to track and actuate virtual and physical articles in coordination with one another. The environment model includes information corresponding to both the physical environment and the corresponding virtual environment” Fisher provides a node that may be implemented as a software device, while Evans provides virtual environment functionality. Combining these would place Evans’ virtual world functionality in one of Fisher’s software nodes, aligning with the claim element.); each node of the plurality of nodes and the operator console node comprising a microprocessor and a memory device, the operator console node further comprising an integrator for integrating the physical world element and the virtual world element(Fisher discloses at para.[0050] “FIG. 3 schematically shows an exemplary embodiment of a node. Each node 210 (or operator console node 215) includes a microprocessor 310 and a memory device 315. The memory device 315 can include or store a main or node process 317 that can include one or more sub- or co-processes 320 that are executable by the microprocessor 310.” Evans then teaches at para[0024] “The server 104 or computing element controls various aspects and components of the system 100, e.g., the server 104 can coordinate operation of modules to generate various physical 120 and/or virtual 118 articles. The server 104 can include a processing element such as a central processing unit (CPU) or a graphics processing unit (GPU) that executes machine readable instructions to generate physical 120 or virtual 118 articles, coordinate communication between, and control, the modules and components of the system 100.” Fisher directly supplies the microprocessor and memory for ever node and the operator console node, while Evans supplies the integration function by coordinating and controlling physical and virtual articles through a combine physical/virtual model.); and wherein the system provides real-time control of the physical world element in response to the integration of the physical world element and the virtual world element(Evans discloses at para[0044] “ The virtual and physical articles may be codependent and varied correspondingly and/or one may be dependent on the other, i.e., the virtual effects will drive the physical effects or vice versa. The coordination and actuation can be based on predicative assessments of either the virtual or physical environments, choreographed or staged articles, or a combination of predictive and choreographed content. In some instances, the environment may also accommodate real time tracking and updating of physical/virtual articles based on un-programmed or planned content. As an example, using the model, light interactions between physical 120 and virtual 118 articles can be extrapolated, e.g., as a virtual article is actuated, the physical articles are actuated based on an estimated impact of the virtual article in the physical environment.” Evans’ combined physical/virtual environment mode provides the integration its real time tracking and updated provides the real-time aspect and actuating the physical article based on the virtual article provides control of the physical world element in response to that integration.) It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Evans into the teachings of Fisher in order to a use Fisher’s established real-time node architecture to implement Evans’ coordinate physical/virtual actuation, providing integrated real-time control of physical and virtual world elements.
Regarding claim 14, the combination of Fisher and Evans disclose all the elements of claim 13 as discussed above. Fisher also discloses further comprising a node process in the memory of a node including instruction executable by the microprocessor to control the physical world element(Fisher states at para.[0006] “Each node of the plurality of nodes includes a microprocessor and a memory device. The memory device includes a node process and at least one process executable by the microprocessor. The at least one process is used to control the operation of the at least one item of equipment.” Further at para.[0050] “ Each node 210 (or operator console node 215) includes a microprocessor 310 and a memory device 315. The memory device 315 can include or store a main or node process 317 that can include one or more sub- or co-processes 320 that are executable by the microprocessor 310.” Further at para.[0063] “In one exemplary embodiment, an axis process can be a software algorithm executed on the microprocessor of a corresponding node to generate instructions to move a motor on a winch to wind or reel.” Here Fisher discloses a node process stored in node memory, including executable instructions run by the node microprocessor to control a physical world element such as a motor/winch.)
Regarding claim 15, the combination of Fisher and Evans disclose all the elements of claim 13 as discussed above. The combination also discloses wherein the integration of the physical world element and the virtual world element includes real-time control of the physical world element includes a response to a condition of the virtual world element. (Evans at para.[0044] “Based on the determined state information for the physical article 120 and/or virtual article 118, as well as the model environment, the method 200 may proceed substantially simultaneously to operations 206 and 208 and the system 100 renders or generates physical articles 120 and virtual articles 118, respectively. In other words, operations 206 and 208 may be executed in parallel. The actuation and selection of particular physical and virtual articles depends on various inputs, frames, timing, or the like, with respect to either the virtual or physical environment. For example, if a new frame in the virtual environment is selected to be rendered at the next moment in time, the corresponding physical article corresponding to the new frame (e.g., light effects based on objects newly presented in the new frame) is selected to be rendered next well.” Evans gives specific examples where a new virtual fram causes the corresponding physical article to be selected and actuated next. That supples the real-time aspect and the causal relationship required by the claim: a condition or change in the virtual world element drives control of the physical world element.) The rationale from claim 13 is incorporated herein.
Regarding claim 16, the combination of Fisher and Evan discloses all the elements of claim 13 as discussed above. Fisher also discloses wherein physical world elements include one or both of a building management system (BMS) and a content management system (CMS)(para.[0044] “Based on the determined state information for the physical article 120 and/or virtual article 118, as well as the model environment, the method 200 may proceed substantially simultaneously to operations 206 and 208 and the system 100 renders or generates physical articles 120 and virtual articles 118, respectively. In other words, operations 206 and 208 may be executed in parallel. The actuation and selection of particular physical and virtual articles depends on various inputs, frames, timing, or the like, with respect to either the virtual or physical environment. For example, if a new frame in the virtual environment is selected to be rendered at the next moment in time, the corresponding physical article corresponding to the new frame (e.g., light effects based on objects newly presented in the new frame) is selected to be rendered next well.” Evans gives specific examples where a new virtual fram causes the corresponding physical article to be selected and actuated next. That supples the real-time aspect and the causal relationship required by the claim: a condition or change in the virtual world element drives control of the physical world element.) The rationale from claim 1 is incorporated herein.
Regarding claim 17, the combination of Fisher and Evans disclose all the elements of claim 13 as discussed above. Fisher also discloses wherein physical world elements may be selected from the group consisting of motors, drivers, cameras, computer vision (CV),sensors, lighting elements, sound/acoustic elements, pyrotechnic elements, video screens, point of sale systems, mobile devices, wearables and combinations thereof(Fisher discloses at para.[0053] “The machinery can be operated by hydraulic, electric, or pneumatic systems. The control system can be used with a variety of different axis devices or nodes that correspond to the controllers for the end machines that make theatrical objects move. Examples of axis machinery can include engines, motors (AC/DC), servos, hydraulic movers, and pneumatic movers.” Fisher teaches items explicitly stated in the claim limitation as well as in a group aligning with the claim language.)
Regarding claim 18, the combination of Fisher and Evans discloses all the elements of claim 13 as discussed above. Evans also discloses wherein virtual world elements may be selected from the group consisting of user profiles, virtual spaces, virtual models,simulators, physics engines, game engines, avatars, applications, big data, artificial intelligence, and combinations thereof(para.[0041] “The mapping is used to generate a model that corresponds to the physical environment and is used to track and actuate virtual and physical articles in coordination with one another. The environment model includes information corresponding to both the physical environment and the corresponding virtual environment.” Further at para.[0048] “Virtual articles 118 can be virtual lights, animations, objects, or other effects that generally would not occur in the physical world. Some examples include: light or energy beams, fluids that appear to defy gravity, animated characters (either fictitious or based on real people), animals, or even simulations like flying, running, swimming, or space travel.” Evans explicitly discloses elements and some combinations of the claim limitations (virtual spaces, models and avatars).) The rationale from claim 1 is incorporated herein.)
Claim 7, which is similar in scope to claim 1, thus rejected under the same rationale.
Claim 8, which is similar in scope to claim 2, thus rejected under the same rationale.
Claim 9, which is similar in scope to claim 3, thus rejected under the same rationale.
Claim 10, which is similar in scope to claim 4, thus rejected under the same rationale.
Claim 11, which is similar in scope to claim 5, thus rejected under the same rationale.
Claim(s) 6,12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fisher as modified by Evans as applied to claim 1 above, and further in view of Toh(US-20220051022-A1).
Regarding claim 6, the combination of Fisher and Evans disclose all the elements of claim 1 as discussed above. However, they do not disclose wherein the integrating includes providing a feature selected from the group consisting of individual wayfinding in a venue, individual content management of a venue system, individual content management of a customer interface, crowd behavior analysis and steering, site-wide control and combinations thereof.
Toh does disclose wherein the integrating includes providing a feature selected from the group consisting of individual wayfinding in a venue,individual content management of a venue system, individual content management of a customer interface, crowd behavior analysis and steering, site-wide control and combinations thereof(para.[0020]“ The technical solutions described herein are related to processing of multiple perception signals to display augmented reality content (e.g., AR objects) for, for example, wayfinding and/or discovery at a venue (e.g., a location, physical space, region, area). Specifically, the accurate positioning and orientation of place and/or event information enables the use of augmented reality displays for use with, for example, wayfinding and/or information discovery. In some implementations, the contextual display in AR assists users in wayfinding at unfamiliar places and/or discovering events or places of interest when on location.” para.[0021] “To achieve accurate placement of AR objects (also can be referred to as points of interest (POIs)), a scale-accurate digital 3D representation of the venue is generated and a location and/or orientation of a user can be localized to the scale-accurate digital 3D representation via AR anchors (e.g., an AR anchor has a fixed location with respect to an origin, wherein the origin is a predefined, fixed location in a real-world physical area).” Para.[0037] “In some implementations, at least one of the AR objects P can be configured to move as the mobile device 110 moves user moves or can move even if the mobile device 110 does not move. For example, one of the AR objects P, such as a navigation guide (e.g., a wayfinding arrow) used to guide a user, can have a starting point near (e.g., at, in front of) a location and orientation of the mobile device 110. As the mobile device 110 moves, the navigation guide can also move (e.g., rotate, move in front of the user) to navigate a user to a desired location.” Para.[0067] “FIGS. 7A and 7B illustrate additional real- world scenes 710, 711 associated with AR objects 701 through 703 for wayfinding within a building (e.g., a specific floor of the building). In this example implementations, the AR objects 701 (an arrow pointing the direction), 702 (a destination marker) can be updated as a user is moving until the user has arrived at the location as shown by AR object 703” These passages disclose a venue being modeled by a real-world physical area, the individual is the particular user device that is localized within that venue and the wayfinding feature is the dynamically updated AR navigation guide that directs the user to a desired destination. Claim 6 is satisfied by any one member of the recited group as such, “individual wayfinding in a venue” is sufficient for the rejection.)
Claim 12, which is similar in scope to claim 6, thus rejected under the same rationale.
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fisher as modified by Evans as applied to claim 13 above, and further in view of Toh(US-20220051022-A1).
Regarding claim 19, the combination of Fisher and Evans disclose all the elements of claim 1 as discussed above. However, they do not disclose wherein the integration of the physical world element and the virtual world element includes providing a feature selected from the group consisting of individual wayfinding in a venue, individual content management of a venue system, individual content management of a customer interface, crowd behavior analysis and steering, site-wide control and combinations thereof.
Toh does disclose wherein the integration of the physical world element and the virtual world element includes providing a feature selected from the group consisting of individual wayfinding in a venue, individual content management of a venue system, individual content management of a customer interface, crowd behavior analysis and steering, site-wide control and combinations thereof(para.[0020]“ The technical solutions described herein are related to processing of multiple perception signals to display augmented reality content (e.g., AR objects) for, for example, wayfinding and/or discovery at a venue (e.g., a location, physical space, region, area). Specifically, the accurate positioning and orientation of place and/or event information enables the use of augmented reality displays for use with, for example, wayfinding and/or information discovery. In some implementations, the contextual display in AR assists users in wayfinding at unfamiliar places and/or discovering events or places of interest when on location.” para.[0021] “To achieve accurate placement of AR objects (also can be referred to as points of interest (POIs)), a scale-accurate digital 3D representation of the venue is generated and a location and/or orientation of a user can be localized to the scale-accurate digital 3D representation via AR anchors (e.g., an AR anchor has a fixed location with respect to an origin, wherein the origin is a predefined, fixed location in a real-world physical area).” Para.[0037] “In some implementations, at least one of the AR objects P can be configured to move as the mobile device 110 moves user moves or can move even if the mobile device 110 does not move. For example, one of the AR objects P, such as a navigation guide (e.g., a wayfinding arrow) used to guide a user, can have a starting point near (e.g., at, in front of) a location and orientation of the mobile device 110. As the mobile device 110 moves, the navigation guide can also move (e.g., rotate, move in front of the user) to navigate a user to a desired location.” Para.[0067] “FIGS. 7A and 7B illustrate additional real- world scenes 710, 711 associated with AR objects 701 through 703 for wayfinding within a building (e.g., a specific floor of the building). In this example implementations, the AR objects 701 (an arrow pointing the direction), 702 (a destination marker) can be updated as a user is moving until the user has arrived at the location as shown by AR object 703” These passages disclose a venue being modeled by a real-world physical area, the individual is the particular user device that is localized within that venue and the wayfinding feature is the dynamically updated AR navigation guide that directs the user to a desired destination. Claim 13 is satisfied by any one member of the recited group as such, “individual wayfinding in a venue” is sufficient for the rejection.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRIS ALEJANDRO PUNTIER whose telephone number is (703)756-1893. The examiner can normally be reached M-F 7:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHRIS ALEJANDRO PUNTIER/Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
