Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 8 is objected to because of the following informalities:
Claim 8 recites a computer readable storage medium. The examiner suggests amending to include a non-transitory computer readable storage medium to clarify that the claimed medium is a tangible storage medium and does not encompass transitory signals.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-6, 8-10, 12-13, 15-17 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Schmirler et al. (US 2018/0131907) in view of Malik (Framework to model virtual factories: a digital twin view, Systems and Control, 2021).
Regarding claim 1, Schmirler et al. (hereinafter Schmirler) discloses a computer-implemented method (Schmirler, [0066], “The one or more processors 320 can perform one or more of the functions described herein with reference to the systems and/or methods disclosed”), comprising:
creating a virtual reality (VR) environment that is based on an industrial floor (Schmirler, [0050], “virtual reality (VR) presentations (referred to collectively herein as “VR/AR presentations”) to a user via a wearable computer or other client device…The system can render a scaled down view of the factory floor area…the system can switch from this external view to an internal view that renders a realistic presentation of the factory floor area from the point of view of a person standing within the environment”);
providing, to at least a first VR device associated with a first user and a second VR device associated with a second user, access to the VR environment (Schmirler, [0158], “To further facilitate coordination of activities between multiple users addressing a detected issue, VR/AR presentation system 302 can allow users to share their personal views with one another. For example, a first user on one side of a machine may wish to share his or her view of the machine with a second user on the other side of the machine or at another location”);
assigning a first gamification role to the first VR device, wherein the first gamification role is based on a first industrial task of the industrial floor (Schmirler, [0111], “information being viewed via a VR/AR presentation can be selectively filtered by the user”. In addition, in paragraph [0112], “user profiles 522 may define the set of information for each machine or device that the user is allowed to view, and rendering component 308 can limit the data that is accessible by the user to those defined sets of data. For example, for users having an “operator” role, rendering component 308 may only allow the user to view data relevant to operation of a machine or automation system (e.g., operating modes, alarm information, running speeds, product counts, etc.)”);
assigning a second gamification role to the second VR device, wherein the second gamification role is based on a second industrial task of the industrial floor (Schmirler, [0111], “information being viewed via a VR/AR presentation can be selectively filtered by the user”. In addition, in paragraph [0112], “user profiles 522 may define the set of information for each machine or device that the user is allowed to view, and rendering component 308 can limit the data that is accessible by the user to those defined sets of data…For users having an “engineering” role, rendering component 308 may further allow the user to view firmware information for control devices, industrial control programming (e.g., ladder logic or other programming), network statistics, or other such engineering data”);
receiving gamification role data from the VR devices (Schmirler, [0156], “monitoring component 316 can also calculate and record performance metrics for the user that rate the user's performance of the workflow, based on the user's measured degree of compliance with or deviation from the workflow. These performance metrics can be based on such factors as a number of detected deviations from the workflow, an average speed at which the user completes workflows, a number of workflows carried out by the user, or other such factors”); and
determining and storing, based on the gamification role data (Schmirler, [0156], “These performance metrics can be recorded in the user profile 522 corresponding to the user”);
though Schmirler teaches a physical industrial floor (Schmirler, [0108], “If the user is physically located within the actual production area, the system allows the user to switch to an AR/VR presentation that renders data, graphics, or virtualized industrial assets over the user's direct view of real industrial assets”. In addition, in paragraph [0166], “This digital twin can be used to test new control programs on virtualized equipment analogous to their real-world counterparts, perform predictive analytics to estimate asset maintenance or replacement schedules, or other such functions”); Schmirler does not expressly discloses “a specification for deploying in the physical industrial floor”;
Malik discloses determining and storing, based on data, a specification for deploying in a physical industrial floor (Malik, 6. Interfacing simulation with physical system, [0001], “With the use of a DT, the robot program is intuitively generated in a simulation environment (Figure 16). Once the desired operation is tested virtually with defined robot trajectories and transition logics the robot program is transferred to the connected cobot that starts working as the robot in the digital twin (offline). An online connection between physical and virtual robot can dynamically transmit any movement made in physical space to the robot in the virtual space and vice versa (Fig. 11)”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to use the concept of Malik’s generating a specification for deploying the robot operation in a physical industrial floor to determine requirements for the physical industrial floor, as taught by Schmirler. The motivation for doing so would have been improving accuracy and efficiency of deployment.
Regarding claim 2, Schmirler discloses the first industrial task is based on a first industrial machine's contribution to a predetermined industrial process within the industrial floor (Schmirler, [0056], “if the user's current view encompasses a real or virtualized motor-driven conveyor and a motor drive that controls the motor, the presentation system may superimpose a current operating status of the motor drive (e.g., a current speed, a fault condition, an operating mode, etc.) near the image or view of the motor drive as perceived by the user”), wherein the second industrial task is based on a second industrial machine's contribution to the predetermined industrial process (Schmirler, [0101], “render subsets of plant data 610, calculated production or machine statistics, or alphanumeric message as overlaid information placed on or near the virtual assets (e.g., control cabinets such as control cabinet 1102, machines, control devices, motors drives, valves, tanks, etc.) to which the information relates”. Fig. 11).
Regarding claim 3, Schmirler discloses generating, a visual simulation of the predetermined industrial process performed by the industrial machines (Schmirler, [0164], “the VR/AR representation of an industrial factory generated by VR/AR presentation system 302 can be used as the basis for a digital twin of the factory. In such embodiments, the plant models 524 can not only model the physical appearance of industrial assets, but can also model certain behaviors of those assets (e.g., responses to control inputs in terms of movement, speed, temperatures, flows, fill levels, etc.). This can allow the VR/AR representation to act as a simulation environment for testing control programs or device configurations”); and
outputting the visual simulation for display on the VR devices (Schmirler, [0164], “a simulation component 318 of the VR/AR presentation system 302 acts as an industrial controller emulator to execute control program 1802 against VR/AR model 1804”).
Schmirler as modified by Malik with the same motivation from claim 1 discloses based on the determined specification (Malik, 6. Interfacing simulation with physical system, [0001], “With the use of a DT, the robot program is intuitively generated in a simulation environment (Figure 16). Once the desired operation is tested virtually with defined robot trajectories and transition logics the robot program is transferred to the connected cobot that starts working as the robot in the digital twin (offline). An online connection between physical and virtual robot can dynamically transmit any movement made in physical space to the robot in the virtual space and vice versa (Fig. 11)”).
Regarding claim 5, Schmirler discloses adding avatars to the VR environment (Schmirler, [0050], “This external view can include real-time avatars representing human operators”), wherein each of the avatars is associated with a different one of the VR devices (Schmirler, [0090], “Returning briefly to FIG. 7, in some embodiments the locations and orientations of the human icons 808a and 808b within the VR/AR presentation can be determined based on location and orientation data 606 received by VR/AR presentation system 302 from the wearable appliances 206 associated with each user”).
Regarding claim 6, Schmirler discloses the avatars are generated based on spatial dimensions associated with the industrial machines (Schmirler, [0094], “other types of devices carried or worn by the human operators and capable of tracking the operators' locations and orientations can also be used to provide VR/AR presentation system 302 with operator location and orientation information”. Fig. 8).
Regarding claim 8, Schmirler discloses a computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable and/or executable by a computer to cause the computer to (Schmirler, [0038], “a component can be, but is not limited to being, a process running on a processor, a processor, a hard disk drive, multiple storage drives (of optical or magnetic storage medium)…a software or a firmware application executed by a processor, wherein the processor can be internal or external to the apparatus and executes at least a part of the software or firmware application”).
The functions recite in claim 8 are similar in scope to the method recited in claim 1 and therefore are rejected under the same rationale.
Regarding claims 9-10, claims 9-10 recite instructions that are similar in scope to the method steps recited in claims 2-3 and therefore are rejected under the same rationale.
Regarding claim 12-13, claims 12-13 recite instructions that are similar in scope to the method steps recited in claims 5-6 and therefore are rejected under the same rationale.
Regarding claim 15, Schmirler discloses a system (Schmirler, [0010], “FIG. 3 is a block diagram of an example virtual and augmented reality presentation system”), comprising:
a processor (Schmirler, [0038], “a processor”); and
logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to ([0038], “a component can be, but is not limited to being, a process running on a processor, a processor, a hard disk drive, multiple storage drives (of optical or magnetic storage medium)…a software or a firmware application executed by a processor, wherein the processor can be internal or external to the apparatus and executes at least a part of the software or firmware application”).
The functions recite in claim 15 are similar in scope to the method recited in claim 1 and therefore are rejected under the same rationale.
Regarding claims 16-17, claims 16-17 recite functions that are similar in scope to the method steps recited in claims 2-3 and therefore are rejected under the same rationale.
Regarding claims 19-20, claims 19-20 recite functions that are similar in scope to the method steps recited in claims 5-6 and therefore are rejected under the same rationale.
Claims 4, 11 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Schmirler et al. (US 2018/0131907) in view of Malik, as applied to claims 2, 9 and 16, in further view of Zhang et al. (Digital twin data‑driven proactive job‑shop scheduling strategy towards asymmetric manufacturing execution decision, Scientific Reports, 2022).
Regarding claim 4, Schmirler as modified by Malik teaches the specification; Schmirler as modified by Malik does not expressly disclose “information that details a plurality of related operational timing sequences of the industrial machines, wherein the related operational timing sequences are based on contributions of the industrial machines with respect to the predetermined industrial process”;
Zhang et al. (hereinafter Zhang) discloses information that details a plurality of related operational timing sequences of the industrial machines (Zhang, Fig. 1 illustrates information that details a plurality of related operational timing sequences of industrial machines), wherein the related operational timing sequences are based on contributions of the industrial machines with respect to predetermined industrial process (Zhang, Local operations delay on makespan, [0002], “In Fig. 1, there is no slack time between the critical operations. When the actual completion time of a critical operation exceeds its theoretical completion time, all subsequent critical operations on the same critical path must be moved backward. Therefore, the delayed completion of critical operations can lead to a delay in makespan”. Each machine contributes a particular operation that forms a portion of the overall process. The timing information identifies when each machine performs its respective operation. The order and timing of the machine operations are determined by industrial process).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to determine the specification of Schmirler as modified by Malik using the proactive scheduling strategy, as taught by Zhang. The motivation for doing so would have been reducing deployment delays and improving efficiency of implementing the specification in the physical industrial environment.
Regarding claim 11, claim 11 recites instruction that is similar in scope to the method step recited in claim 4 and therefore is rejected under the same rationale.
Regarding claim 18, claim 18 recites function that is similar in scope to the method step recited in claim 4 and therefore is rejected under the same rationale.
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Schmirler et al. (US 2018/0131907) in view of Malik, as applied to claims 1 and 8, in further view of Tsirkin (US 2021/0266148).
Regarding claim 7, Schmirler as modified by Malik does not expressly disclose “implemented in a trusted execution environment (TEE), wherein the TEE is a secure container”;
Tsirkin discloses a trusted execution environment (TEE), wherein the TEE is a secure container (Tsirkin, [0014], “Launching a TEE instance, such as a trusted container, by a cloud provider typically involves using a secret that is private to the TEE instance owner and unavailable to the cloud provider”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to implement the industrial visualization system of Schmirler using a trusted execution environment, as taught by Tsirkin. The motivation for doing so would have been providing a secure and isolated execution environment for processing and displaying industrial visualization information.
Regarding claim 14, claim 14 recites instruction that is similar in scope to the method step recited in claim 7 and therefore is rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYLE ZHAI whose telephone number is (571)270-3740. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ke Xiao can be reached at (571) 272 - 7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KYLE ZHAI/ Primary Examiner, Art Unit 2611
Prosecution Insights