Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 24,25,27-29,31,33,35,37-40,42,44-45,47-57 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20150370661 A1 (Swanson) in view of US 20190129814 A1 (Tati) and US 20190149399 A1 (Reed).
Regarding claim 24, Swanson teaches,
A method of performing input/output (IO) domain failover in a fault tolerant computer system having a plurality of physical CPU node sand a plurality of physical IO domains (fig 1a: node1, node2; par 32 “Generally, reconfiguration of various components and interfaces on a multi-node PCH may be implemented via switching circuitry and control logic embedded on the PCH or a different chipset interconnect from the multi-node PCH to a CPU I/O interface. Moreover, the multi-node PCH chipset, with System Management Mode (SMM) support, can dynamically change the links while preserving system context and not requiring an Operating System (OS) reset. Additionally, support for this functionality does not require any OS changes and provides self-healing for any component in the MN-PCH required for standard OS survivability and support. Also, it provides resilience for other non-critical components in the MN-PCH, include both high-speed and legacy interfaces and controllers.”); the method comprising:
designating one of the CPU nodes a standby CPU node and designating one or more of the other CPU nodes as active CPU nodes;(fig 1a; par 30 “For example, in one embodiment, MNPCH 102 and MN-PCH 102a include the same components, interconnects, and logic, while MN-PCH 102 is configured to support four nodes and MN-PCH 102a is configured to support two nodes under which the (now) unused components and interconnects for the removed nodes (Node3 and Node 4) are used as spares configured to replace primary components and interconnects via associated failover operations.”), wherein each CPU node comprises a processor and a memory, wherein one or more of the active CPU nodes is executing an operating system and one or more customer applications(fig 4:408,406,”operating system”, “software applications”; par 49 “Each of Node1 and Node2 include a processor 404 and local system memory 406.”; par 52 “Subsequently, software components comprising the operating system instance will be loaded into a protected region of memory 404. After the OS has booted, software components used for software applications may be loaded into a user space region of memory 406 allocated for applications by the OS.”)
designating at least one of the active IO domains as a primary IO domain, wherein each IO domain of the plurality of IO domains comprises an IO board (fig 2a:214,214S; par 31 “In a similar manner, HSIO controllers 214 for Node 3 and Node 4 of MN-PCH 102 have been reconfigured as spare HSIO controllers 214s for each of Node1 and Node2 for MN-PCH 102a.”; par 26 “Moreover, the number of lanes used for primary HSIO controllers may be different than the number of lanes used for spare HSIO controllers, as described and illustrated below.”);
performing active communication functions using the primary IO domain for the active CPU nodes (par 31 “As illustrated in FIGS. 1a and 2a, DMI links 104, 105, 106, and 107 in MN-PCH 102 are reconfigured in MNPCH 102a to provide a primary DMI link and interface and a spare DMI link and interface for each of Node1 and Node2. After reconfiguration, these links and interfaces are depicted as primary DMI links 104p and 106p, primary DMI interfaces 104ip and l06ip, spare DMI links 104s and 106s, and spare DMI interfaces l04ip and l06ip.”), wherein each CPU node of the plurality of CPU nodes comprises a processor and a memory(par 49 “Each of Node1 and Node2 include a processor 404 and local system memory 406.”)
connecting each CPU node to each IO domain using a switching fabric, wherein each IO domain is connected to the switching fabric by one or more links;(par 135 “a plurality of sets of high-speed Input-Output (IO) controllers, coupled to the high-speed root fabric, each configurable to be dedicated for use by a respective node, …”)
enabling a failure trigger for each switching fabric control component in each IO domain to detect a failure of the primary or secondary IO domain or a component thereof, wherein the failure trigger comprises one or more of link-down errors, uncorrectable and fatal errors, and software triggers;(fig 3a:306; par 36 “In response to the asserted error, a determination is made in a decision block 306 to whether the error is fatal or otherwise uncorrectable.”)
upon failure trigger occurring, stopping drivers using the failing IO domain or failing IO component (par 41 “Some processors support a quiesce mode under which interfaces and other components are quiesced (e.g., put into a sleep mode)…. the flow proceeds to a block 338 in which the quiesce mode is initiated in SMI to ensure all I/O traffic is stopped.”; fig 3c; par 44 “In a decision block 354 a determination is made to whether the failover is a controller failover. If the answer is YES, the flow proceeds to a block 354 in which the failing controller is disabled and the failover controller (e.g., the spare controller) is enabled. The interconnect hierarchy is then reconfigured in a block 356, resulting in the chipset being healed, as depicted in a block 358.”); and
continuing to use the remaining operational IO domains and IO components, (fig 3c; par 44 “In a decision block 354 a determination is made to whether the failover is a controller failover. If the answer is YES, the flow proceeds to a block 354 in which the failing controller is disabled and the failover controller (e.g., the spare controller) is enabled. The interconnect hierarchy is then reconfigured in a block 356, resulting in the chipset being healed, as depicted in a block 358.”), and
if one of: a failure, a beginning of a failure and a predicted failure occurs in an active node(fig 3a:304,306; par 35 “In a block 304 an error is detected and asserted. There are various well-known mechanisms for asserting errors on platforms,…. Similarly, there are well-known mechanisms for detecting errors.” par 45 “In a block 364 a determination is made to whether the node or a node controller has failing or has failed. If the node or controller is detected as failing/failed, the answer to decision block 364 is YES, and the flow proceeds to a block 366 in which the failing/failed controller is disabled and the failover controller is enabled.”), transferring current processor state of the active CPU node to the standby CPU node,(par 34 “When SMM is invoked through an SMI, the processor (e.g., Node CPU) saves its current state and switches to a separate operating environment … When the SMI handler has completed its operations, it executes a resume instruction. This instruction causes the processor to reload its saved state, switch back to protected or real mode, and resume executing the interrupted application or OS tasks.” Once the failing CPU Node has saved its operating state, the failing node is disabled and replaced with the failover node(explained in fig 3a:366; par 45 “the flow proceeds to a block 366 in which the failing/failed controller is disabled and the failover controller is enabled.”), which becomes the new active node, then the healed(new active node) processor reloads the failing node’s saved state and continues the interrupted application and operating system tasks.)
wherein the current processor state comprise the state of the running operating system and the one or more customer applications such that the operating system and the one or more customer applications are transferred to the standby CPU node(fig 3c:366,370,374; par 34 “When SMM is invoked through an SMI, the processor (e.g., Node CPU) saves its current state and switches to a separate operating environment … When the SMI handler has completed its operations, it executes a resume instruction. This instruction causes the processor to reload its saved state, switch back to protected or real mode, and resume executing the interrupted application or OS tasks.”;),
wherein the standby CPU node becomes new active CPU node; and continue executing, using the new active CPU node, the transferred operating system and the one or more customer applications.(par 45 “In a block 364 a determination is made to whether the node or a node controller has failing or has failed. If the node or controller is detected as failing/failed, the answer to decision block 364 is YES, and the flow proceeds to a block 366 in which the failing/failed controller is disabled and the failover controller is enabled.”, par 46 “The flow next proceeds from either block 366 or 368 to a block 370 in which the internal decode path is reconfigured. Following this, the DMI link is returned to a normal power and operating state”. Once the failing CPU Node has saved its operating state, the failing node is disabled and replaced with the failover node, which becomes the new active node, then the healed(new active node) processor reloads the failing node’s saved state and continues the interrupted application and operating system tasks.)
However, although Swanson teaches saving the current processor state of the failing active CPU node and resuming from the saved state on the standby node, Swanson does not specifically teach transferring memory contents of the active CPU node to the standby CPU node.
On the other hand, Tati teaches
A method of providing a fault tolerant computer system having a plurality of physical CPU nodes; the method comprising: (Par 9 “Techniques for implementing application fault tolerance via battery-backed replication of volatile state are provided. In one set of embodiments, a primary host system can detect a failure that causes an application of the primary host system to stop running. In response to detecting the failure, the primary host system can replicate volatile state that is used by the application to a secondary host system, where the secondary host system maintains a copy of the application, and where execution of the application is failed over to the copy on the secondary host system using the replicated volatile state.”; fig 2; par 27)
designating one of the CPU nodes a standby CPU node and designating one or more of the other CPU nodes as active CPU nodes(fig 2:104”primary host system”, “secondary host system”; par 27 “As shown in FIG. 2, a novel post-fail agent 204 implemented in OS 108 or the BIOS of primary host system 102 can, upon the occurrence of a host failure, replicate the contents of application volatile state 120 to a corresponding agent 206 on secondary host system 104.”), wherein each CPU node comprises a processor and a memory, wherein one or more of the active CPU nodes is executing an operating system and one or more customer applications;(fig 2:110,108,122; par 21 “As shown, architecture 100 includes a primary host system 102, a secondary host system 104, …. Primary host system 102 includes an OS 108 and an FT-protected application 110 …. Secondary host system 104 includes an OS 112 and a copy, or replica, 114 of application 110. OSs 108 and 112 can be conventional "bare-metal" OSs ( e.g., Linux, Windows, etc.) or virtualization hypervisors (e.g., ESXi, Hyper-V, etc.). Similarly, application 110 can be a conventional software application or a virtual machine (VM).; par 22 “… FT managers 116 and 118 can coordinate to migrate the volatile state of application 110 (shown via reference numeral 120 in volatile memory 122) from primary host system 102 to secondary host system 104, thereby creating an initial instance of secondary copy 114 and corresponding volatile state 124 in volatile memory 126.”; par 25)
wherein each CPU node of the plurality of CPU nodes comprises a processor and a memory;(par 25 “assumes that primary host system 102 is modified to include a battery 202 that can power critical components of system 102 ( e.g., CPU, volatile memory, network interface) in cases where system 102 loses AC power.”)
if one of: a failure, a beginning of a failure and a predicted failure occurs in an active node(fig 3:302; par 36 “Starting with block 302, primary host system 102 can detect the occurrence of a failure that prevents application 110 from continuing its execution. Examples of such failures include, e.g., a loss of AC power, an OS-level crash, a "hard" crash caused by, e.g., an uncorrectable memory error, and a system shutdown initiated by a hard press of the system's power button.”), transferring processor state and memory contents of the active CPU node to the standby CPU node.(fig 3:310; par 38 “Upon being invoked, post-fail agent 204 can take a checkpoint of volatile state 120 of application 110 and copy the entire contents of this state (as captured in the check point) to agent 206 of secondary host system 104 (block 310). Volatile state 120 can include, e.g., the volatile memory allocated to application 110, as well as other non-persistent information pertaining to application 110 (e.g., virtual device state if application 110 is a VM).”)
wherein the processor state and memory contents comprise the state of the running operating system and the one or more customer applications such that the operating system and the one or more customer applications are transferred to the standby CPU node(fig 3:310; par 38 “Upon being invoked, post-fail agent 204 can take a checkpoint of volatile state 120 of application 110 and copy the entire contents of this state (as captured in the check point) to agent 206 of secondary host system 104 (block 310). Volatile state 120 can include, e.g., the volatile memory allocated to application 110, as well as other non-persistent information pertaining to application 110 (e.g., virtual device state if application 110 is a VM).”),
wherein the standby CPU node becomes new active CPU node; and continue executing, using the new active CPU node, the transferred operating system and the one or more customer applications.(fig 3:314; par 40 “Finally, once the entirety of volatile state 120 has been successfully copied to secondary host system 104, post-fail agent 204 can send a signal to agent 206 indicating that the application can be failed over to secondary host system 104 (block 314) and workflow 300 can end.”)
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify Swanson to incorporate the transferring memory contents of the active CPU node to the standby CPU node of Tati. One of ordinary skill in the art would have been motivated to remedy the shortcomings of Swanson -- a need for a solution for the issue of how to smoothly resume operations on the failover host(Tati par 7 “A significant advantage of this approach is that it has no impact on the application's performance during normal operation. However, the application's volatile state on the original host system is lost during the fail-over and is recreated on the new host system; thus, any non-committed transactions at the point of failure are aborted. Further, the fail-over time for HA (which requires restarting the application on the new host system) is not instantaneous and can potentially take several seconds or minutes.”) -- with Tati providing a known method to solve a similar problem. Tati provides “Techniques for implementing application fault tolerance via battery-backed replication of volatile state are provided. In one set of embodiments, a primary host system can detect a failure that causes an application of the primary host system to stop running. In response to detecting the failure, the primary host system can replicate volatile state that is used by the application to a secondary host system, where the secondary host system maintains a copy of the application, and where execution of the application is failed over to the copy on the secondary host system using the replicated volatile state.”(Tati par 9)
However, Swanson and Tati do not specifically teach wherein each physical CPU node and physical IO domain are configured to be removed and replaced without affecting the one or more customer applications executing on one or more of the other physical CPU nodes and physical IO domains.
On the other hand, Reed teaches,
A method of performing dynamic reconfiguration in a fault tolerant computer system (provisional par 250 “Described above are example details regarding Riptide, which divides the TS hyper-kernel into two layers, one dealing with implementation of a single scalable virtual Intel hardware system, and a lower layer that provides abstractions of Logical Modules used to implement all of the functions needed in a way that allows for dynamic reconfiguration while TidalPod continues to run.”; ) having a plurality of physical CPU nodes and a plurality of physical IO domains(provisional par 100 “Various examples of modules in a TidalPod will be described in further detail below. Some examples include: a node ( one or more processor chips, a motherboard, its power supply, and a number of memory DIMMs, etc.), the interconnect network (the switches, cables, and NI Cs that provide inter-node connectivity),”; provisional par 242 “Riptide's framework achieves this by using multiple ports on each node and multiple switches, such that the failure of a switch or a cable merely degrades performance, but an alternate delivery path always exists”; Switches and cables are physical components represented by a logical module; provisional par 132 ” In some embodiments, a BusPort is the logical module that represents a highspeed interconnection from a Node to the other logical nodes attached to an Internet switch. In some embodiments, there is one held in each operational node.”; provisional par 245,246,247); the method comprising:
designating one of the CPU nodes a standby CPU node and designating one or more of the other CPU nodes as active CPU nodes, wherein each CPU node comprises a processor and a memory, (provisional par 80” Considering, for example, that since resources are all mobile, previously configured hot standby machines can be utilized. When failure is suspected to occur in the near future due, for example, to soft ECC errors, server temperature rising, or higher than normal network anomalies, it may be dealt with, for example, by dynamically adding an additional hot standby node to the cluster, informing all nodes about the pending node failure so they do not migrate any virtual processors to it, and having the failing node evict virtual processors at the earliest possible time, and pages of memory in active use.”) wherein one or more of the active CPU nodes is executing an operating system and one or more customer applications;(provisional par 86 “As used herein, dynamic reconfiguration refers to the capability of changing the set of hardware components contained in a TidalPod while the system continues to run a guest operating system and its applications.”)
wherein each physical CPU node and physical IO domain are configured to be removed and replaced (provisional par 86 “As used herein, dynamic reconfiguration refers to the capability of changing the set of hardware components contained in a TidalPod while the system continues to run a guest operating system and its applications.”; provisional par 80; provisional par 250; provisional par 97 “Dynamic reconfiguration in the Riptide design or implementation has an additional benefit for resiliency. Physical devices may be taken out of service for repair, replacement, or upgrades, while the TidalPod and its guest operating system continues to run. Replaceable devices can be removed from the TidalPod without disrupting it. Moreover, they may remain in place for an arbitrary period until a removal for replacement or upgrade can be scheduled. In many datacenter environments, such flexibility can be quite valuable - one both reduces downtime by continuing to run, and at the same time allows the repair to be done right, without need for "emergency on-call specialists".)without affecting the one or more customer applications executing on one or more of the other physical CPU nodes and physical IO domains, (provisional par 86 “As used herein, dynamic reconfiguration refers to the capability of changing the set of hardware components contained in a TidalPod while the system continues to run a guest operating system and its applications.”;)
enabling a failure trigger for each component in each domain to detect a failure of the component,(provisional par 77 “a) early indicators signal a possible impending hardware failure,”; provisional par 80 “Considering, for example, that since resources are all mobile, previously configured hot standby machines can be utilized. When failure is suspected to occur in the near future due, for example, to soft ECC errors, server temperature rising, or higher than normal network anomalies, it may be dealt with, for example, by dynamically adding an additional hot standby node to the cluster, informing all nodes about the pending node failure so they do not migrate any virtual processors to it, and having the failing node evict virtual processors at the earliest possible time, and pages of memory in active use.”; provisional par 426, par 427)
wherein the failure trigger comprises one or more of link-down errors, uncorrectable and fatal errors, and software triggers;( provisional par 91 “… the distribution of failures among component types is typically extremely skewed. CPUs, for example, almost never fail, while network cables and switches fail quite frequently, power supplies overheat, fans fail, etc.”; provisional par 242 “Riptide's framework achieves this by using multiple ports on each node and multiple switches, such that the failure of a switch or a cable merely degrades performance, but an alternate delivery path always exists”; provisional par 243 “…selection of paths and detection of failed paths is done in the network kernel driver…” )
upon failure trigger occurring, stopping drivers using the failing IO domain or failing IO component;( provisional par 242 “In order to make the Interconnect among nodes resilient, redundancy is added. In some embodiments, Riptide's framework achieves this by using multiple ports on each node and multiple switches, such that the failure of a switch or a cable merely degrades performance, but an alternate delivery path always exists.”; provisional par 132 defines what a BusPort is ” In some embodiments, a BusPort is the logical module that represents a highspeed interconnection from a Node to the other logical nodes attached to an Internet switch. In some embodiments, there is one held in each operational node.”; and provisional par 245,246,247 go into more detail about working logical nodes that are added and failing logical nodes that are removed inside the BusPort logical module.)
continuing to use the remaining operational IO domains and IO components;( provisional par 242 “In order to make the Interconnect among nodes resilient, redundancy is added. In some embodiments, Riptide's framework achieves this by using multiple ports on each node and multiple switches, such that the failure of a switch or a cable merely degrades performance, but an alternate delivery path always exists.”; provisional par 132(defines BusPort); provisional par 245,246,247(describes the logical node failover options))
if one of: a failure, a beginning of a failure and a predicted failure occurs in an active node,(provisional par 242 “Riptide's framework achieves this by using multiple ports on each node and multiple switches, such that the failure of a switch or a cable merely degrades performance, but an alternate delivery path always exists.”; provisional par 243 “This implies that selection of paths and detection of failed paths is done in the network kernel driver.”; provisional par 77 “a) early indicators signal a possible impending hardware failure,”; provisional par 80 “Considering, for example, that since resources are all mobile, previously configured hot standby machines can be utilized. When failure is suspected to occur in the near future due, for example, to soft ECC errors, server temperature rising, or higher than normal network anomalies, it may be dealt with, for example, by dynamically adding an additional hot standby node to the cluster, informing all nodes about the pending node failure so they do not migrate any virtual processors to it, and having the failing node evict virtual processors at the earliest possible time, and pages of memory in active use.”; provisional par 426, par 427) transferring computation state of the active CPU node to the standby CPU node,( provisional par 428 “Continuations are built using the "InitContinuation" routine. If a decision is made to move the computation, the remote physical node holding the resource will build a continuation that corresponds to the stalled computation and will store it in the remote physical node's event table. When that continuation resumes, the resource will be available. In effect, the hyperkernel has transferred the virtual processor to a different node.”; provisional par 513 “In the event an addresses needs to be put into a continuation, care is taken in the move, since the address is a physical address of the source, and bears no relationship with the physical address in the destination. Moving a continuation means copying its contents to the destination node as discussed above, and remapping any physical addresses from the source to the target.”)
wherein the computation state comprise the state of the running operating system and the one or more customer applications such that the operating system and the one or more customer applications are transferred to the standby CPU node, wherein the standby CPU node becomes new active CPU node;(provisional par 428 “Continuations are built using the "InitContinuation" routine. If a decision is made to move the computation, the remote physical node holding the resource will build a continuation that corresponds to the stalled computation and will store it in the remote physical node's event table. When that continuation resumes, the resource will be available. In effect, the hyperkernel has transferred the virtual processor to a different node.”; provisional par 513 “In the event an addresses needs to be put into a continuation, care is taken in the move, since the address is a physical address of the source, and bears no relationship with the physical address in the destination. Moving a continuation means copying its contents to the destination node as discussed above, and remapping any physical addresses from the source to the target.”) and
continue executing, using the new active CPU node, the transferred operating system and the one or more customer applications.(provisional par 86 “As used herein, dynamic reconfiguration refers to the capability of changing the set of hardware components contained in a TidalPod while the system continues to run a guest operating system and its applications.”; provisional par 97 “Dynamic reconfiguration in the Riptide design or implementation has an additional benefit for resiliency. Physical devices may be taken out of service for repair, replacement, or upgrades, while the TidalPod and its guest operating system continues to run. Replaceable devices can be removed from the TidalPod without disrupting it.”; provisional par 972 “From the guest operating system's point of view, it did not observe a fault (the hyper-kernel intercepted the faults and performed the vm exit/enter). Instead, the guest operating system attempted to access a page, and at the next instruction, it has already accessed the page (where the guest operating system is not aware of the underlying migration that was performed by the hyper-kernel).”;)
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify Swanson and Tati to incorporate CPU node and IO domain are configured to be removed and replaced without affecting applications executing on one or more of the other CPU nodes and IO domains of Reed. One of ordinary skill in the art would have been motivated to remedy the shortcomings of Swanson and Tati -- a need for how to replace components after they have failed(Reed provisional par 91 “It may not be possible to build a system that never fails. However, years of experience have taught the IT industry that the distribution of failures among component types is typically extremely skewed. CPUs, for example, almost never fail, while network cables and switches fail quite frequently, power supplies overheat, fans fail, etc.”) -- with Reed providing a known method to solve a similar problem. Reed provides “As described herein, Riptide provides a framework for exploiting standard redundant hardware techniques that drastically reduce the impact of failure of the components that dominate the causes of downtime. Using the techniques described herein. hardware failures are transformed into degradation of performance, which can then be dealt with by dynamic reconfiguration. This type of approach provides an arbitrary number of"9's" of availability.”(Reed provisional par 91)
Regarding claim 25, Swanson, Tati, and Reed teaches,
The method of claim 2,
Swanson further teaches,
further comprising isolating the failing IO domain or IO component upon failure trigger occurring. (fig 3c; par 44 “In a decision block 354 a determination is made to whether the failover is a controller failover. If the answer is YES, the flow proceeds to a block 354 in which the failing controller is disabled and the failover controller (e.g., the spare controller) is enabled. The interconnect hierarchy is then reconfigured in a block 356, resulting in the chipset being healed, as depicted in a block 358.”).
Regarding claim 27, Swanson and Tati teaches,
The method of claim 24,
Swanson further teaches,
further comprising generating an interrupt for each CPU node, wherein the interrupt communicates that the IO domain or component there has failed(fig 3a:326; par 39 “If the answer to decision block 322 is NO, in one embodiment a sideband interrupt is asserted to interrupt the CPU complex to initiate the SMI.”) to each CPU node.( Fig 3c:356; par 44 “If the answer is YES, the flow proceeds to a block 354 in which the failing controller is disabled and the fail over controller ( e.g., the spare controller) is enabled. The interconnect hierarchy is then reconfigured in a block 356, resulting in the chipset being healed, as depicted in a block 358.” The interconnect hierarchy is how each CPU node knows how to connect to an IO domain.)
Regarding claim 28, Swanson, Tati, and Reed teaches,
The method of claim 24,
Swanson further teaches,
wherein each IO domain further comprises a set of IO devices (fig 2a:214,214S; par 25 “In the illustrated embodiment, each HSIO controller 214 includes a PCie controller 216, a SATA controller 218, and a USB controller 220. In addition to the HSIO controllers shown, other types of high-speed controllers and/or interfaces may also be provided, such as an extensible host controller interface (XHCI). Moreover, an HSIO controller may include one or more instances of an associated interface (e.g., multiple PCie, SATA, and/or USB interfaces).”), wherein the set of IO devices comprises the component of the primary IO domain (fig 2a:214; par 29,30,31 “For illustrative purposes, spare interfaces and components are depicted in light gray to differentiate them from primary components.”…” In a similar manner, HSIO controllers 214 for Node 3 and Node 4 of MN-PCH 102 have been reconfigured as spare HSIO controllers 214s for each of Node1 and Node2 for MN-PCH 102a.”), wherein each IO device comprises one or more physical functions and/or virtual functions, wherein one or more physical and/or virtual functions in one IO domain are shareable.( fig 2a:214,214S; par 25 “In the illustrated embodiment, each HSIO controller 214 includes a PCie controller 216, a SATA controller 218, and a USB controller 220.”)
Regarding claim 29, Swanson, Tati, and Reed teaches,
The method of claim 24,
Swanson further teaches,
wherein each CPU node and IO domain are configured to be replaced without affecting the one or more customer applications executing on one or more of the other CPU nodes and IO domain (par 32 “Generally, reconfiguration of various components and interfaces on a multi-node PCH may be implemented via switching circuitry and control logic embedded on the PCH or a different chipset interconnect from the multi-node PCH to a CPU I/O interface. Moreover, the multi-node PCH chipset, with System Management Mode (SMM) support, can dynamically change the links while preserving system context and not requiring an Operating System (OS) reset. Additionally, support for this functionality does not require any OS changes and provides self-healing for any component in the MN-PCH required for standard OS survivability and support. Also, it provides resilience for other non-critical components in the MN-PCH, include both high-speed and legacy interfaces and controllers.”; par 55)
Regarding claim 31, Swanson, Tati, and Reed teaches,
The method of claim 24
Swanson further teaches,
further comprising allocating the set of IO devices and the one or more physical and/or virtual functions to one or more CPU nodes and one or more of two switching fabric control components. (fig 2a:214,214S; par 25 “In the illustrated embodiment, each HSIO controller 214 includes a PCie controller 216, a SATA controller 218, and a USB controller 220. In addition to the HSIO controllers shown, other types of high-speed controllers and/or interfaces may also be provided, such as an extensible host controller interface (XHCI). Moreover, an HSIO controller may include one or more instances of an associated interface (e.g., multiple PCie, SATA, and/or USB interfaces).”)
Regarding claim 33, Swanson, Tati, and Reed teaches,
The method of claim 24,
Swanson further teaches,
wherein each IO domain further comprises a set of IO devices, the method further comprising partitioning one or more of the set of IO devices and the virtual functions among a set of physical CPU nodes (par 24 “In one embodiment, MN-PCH 102 supports dedicated platform level signals per node for per-node power delivery control. As a result, individual nodes can execute state transitions and host partition resets without impacting other nodes.”), the set of physical CPU nodes comprising the active CPU node and the standby CPU node(claim 9 “The MN-PCH of claim 2, wherein the MN-PCH may be selectively configured to implement PCH facilities for: N nodes, each node including at least one central processing unit (CPU) or CPU complex; and N/2 nodes with redundancy, wherein PCH facilities for the N/2 nodes that are not used are reconfigured as spare PCH facilities for the N/2 nodes coupled to the MNPCH.”; par 85 )
Regarding claim 35, Swanson, Tati, and Reed teaches,
The method of claim 24
Swanson further teaches,
further comprising controlling communication through a management processor of an active IO domain through the switching fabric. (par 26 “The High-Speed IO capability (e.g., provided by PCie controller 216, SATA controller 218, and USB controller 220) of MN-PCH 102 is highly configurable amongst a set of generic physical lanes. The lanes can be flexibly mapped to various integrated IO controllers (not shown). In addition, each controller can be configured to any node during the initial boot sequence. This allows the customer to partition the IO capabilities amongst the nodes supported in the system.”)
Regarding claims 37,38,56 they are the system implementing the method of claims 29,25,54 and are rejected for the same reasons.
Regarding claim 39, Swanson, Tati, and Reed teaches,
The system of claim 37,
Swanson further teaches,
wherein if one of a failure, a beginning of a failure, and a predicted failure occurs in an active IO domain, the active IO domain is deprovisioned and a secondary IP domain is used in lieu thereof. (fig 3c; par 44 “In a decision block 354 a determination is made to whether the failover is a controller failover. If the answer is YES, the flow proceeds to a block 354 in which the failing controller is disabled and the failover controller (e.g., the spare controller) is enabled. The interconnect hierarchy is then reconfigured in a block 356, resulting in the chipset being healed, as depicted in a block 358.”).
Regarding claim 40, it is the system implementing the method of claim 25 and is rejected for the same reasons.
Regarding claim 42, it is the system implementing the method of claim 35 and is rejected for the same reasons.
Regarding claims 44,45 they are the system implementing the method of claims 27,28 and are rejected for the same reasons.
Regarding claim 47, Swanson, Tati, and Reed teaches,
The system of claim 37,
Swanson further teaches,
wherein each IO domain comprises at least two switching fabric control components, each switching fabric control component in communication with the switching fabric. (fig 2a:216,218,200; par 25 “In the illustrated embodiment, each HSIO controller 214 includes a PCie con troller 216, a SATA controller 218, and a USB controller 220. In addition to the HSIO controllers shown, other types of high-speed controllers and/or interfaces may also be provided, such as an extensible host controller interface (XHCI). Moreover, an HSIO controller may include one or more instances of an associated interface (e.g., multiple PCie, SATA, and/or USB interfaces).”)
Regarding claim 48, Swanson, Tati, and Reed teaches,
The method of claim 24,
Swanson further teaches,
wherein the memory contents and processor state comprise the state of the running operating system and the one or more customer applications such that the operating system and the one or more customer applications are transferred to the standby CPU node.(fig 3c:366,370,374; par 34 “When SMM is invoked through an SMI, the processor (e.g., Node CPU) saves its current state and switches to a separate operating environment … When the SMI handler has completed its operations, it executes a resume instruction. This instruction causes the processor to reload its saved state, switch back to protected or real mode, and resume executing the interrupted application or OS tasks.”; par 45 “In a block 364 a determination is made to whether the node or a node controller has failing or has failed. If the node or controller is detected as failing/failed, the answer to decision block 364 is YES, and the flow proceeds to a block 366 in which the failing/failed controller is disabled and the failover controller is enabled.”, par 46 “The flow next proceeds from either block 366 or 368 to a block 370 in which the internal decode path is reconfigured. Following this, the DMI link is returned to a normal power and operating state”. Once the failing CPU Node has saved its operating state, the failing node is disabled and replaced with the failover node, which becomes the new active node, then the healed(new active node) processor reloads the failing node’s saved state and continues the interrupted application and operating system tasks.)
Regarding claim 49, Swanson, Tati, and Reed teaches,
The system of claim 37,
Swanson further teaches,
wherein each CPU node and each IO domain are a modular component, wherein each modular component may be replaced, if failing(par 29 “In some embodiments, a multi-node PCH may be reconfigured to support redundant capabilities to increase reliability. For example, such a multi-node PCH may employ the redundant capabilities to support dynamic node healing under which the MN-PCH can be dynamically reconfigured such that a failed or failing primary interface or component is replaced with a corresponding spare interface or component.”), without affecting a corresponding non-failing modular component. (Par 32 “Moreover, the multi-node PCH chipset, with System Management Mode (SMM) support, can dynamically change the links while preserving system context and not requiring an Operating System (OS) reset. Additionally, support for this functionality does not require any OS changes and provides self-healing for any component in the MN-PCH required for standard OS survivability and support. Also, it provides resilience for other non-critical components in the MN-PCH, include both high-speed and legacy interfaces and controllers.”)
Also, Reed further teaches,
wherein each modular component may be removed and replaced, if failing, without affecting a corresponding non-failing modular component.(provisional par 97 “Dynamic reconfiguration in the Riptide design or implementation has an additional benefit for resiliency. Physical devices may be taken out of service for repair, replacement, or upgrades, while the TidalPod and its guest operating system continues to run. Replaceable devices can be removed from the TidalPod without disrupting it.”)
Regarding claim 50, Swanson, Tati, and Reed teaches,
The system of claim 37,
Swanson further teaches,
wherein if one of: a failure, a beginning of a failure and a predicted failure occurs in an active node(fig 3a:304,306; par 35 “In a block 304 an error is detected and asserted. There are various well-known mechanisms for asserting errors on platforms,…. Similarly, there are well-known mechanisms for detecting errors.” par 45 “In a block 364 a determination is made to whether the node or a node controller has failing or has failed. If the node or controller is detected as failing/failed, the answer to decision block 364 is YES, and the flow proceeds to a block 366 in which the failing/failed controller is disabled and the failover controller is enabled.”), processor state and memory contents of the active CPU node is transferred to the standby CPU node, wherein the memory contents and processor state comprise the state of the running operating system and the one or more customer applications such that the operating system and the one or more customer applications are transferred to the standby CPU node, wherein the standby CPU node becomes a new active CPU node. (fig 3c:366,370,374; par 34 “When SMM is invoked through an SMI, the processor (e.g., Node CPU) saves its current state and switches to a separate operating environment … When the SMI handler has completed its operations, it executes a resume instruction. This instruction causes the processor to reload its saved state, switch back to protected or real mode, and resume executing the interrupted application or OS tasks.”; par 45 “In a block 364 a determination is made to whether the node or a node controller has failing or has failed. If the node or controller is detected as failing/failed, the answer to decision block 364 is YES, and the flow proceeds to a block 366 in which the failing/failed controller is disabled and the failover controller is enabled.”, par 46 “The flow next proceeds from either block 366 or 368 to a block 370 in which the internal decode path is reconfigured. Following this, the DMI link is returned to a normal power and operating state”. Once the failing CPU Node has saved its operating state, the failing node is disabled and replaced with the failover node, which becomes the new active node, then the healed(new active node) processor reloads the failing node’s saved state and continues the interrupted application and operating system tasks.)
Regarding claim 51, Swanson, Tati, and Reed teaches,
The method of claim 24,
Swanson further teaches,
wherein each CPU node(fig 4a:Node1, Node2; par 48 “In the illustrated embodiment of FIG. 4a, system architecture 400 includes a pair of nodes labeled Node1 and Node2, as before. For illustrative purposes, Node2 is depicted as a simple block, while Node1 is shown in further detail;”) and each IO domain are a modular component(fig 2:”HSIO N1”,”HSIO N2”,”HSIO N3”, “HSIO N4”; par 31 “In a similar manner, HSIO controllers 214 for Node 3 and Node 4 of MN-PCH 102 have been reconfigured as spare HSIO controllers 214s for each of Nadel and Node2 for MN-PCH 102a.”), wherein each modular component may be replaced, if failing, without affecting a corresponding non-failing modular component. (par 24 “In one embodiment, MN-PCH 102 supports dedicated platform level signals per node for per-node power delivery control. As a result, individual nodes can execute state transitions and host partition resets without impacting other nodes.”; par 29 “In some embodiments, a multi-node PCH may be reconfigured to support redundant capabilities to increase reliability. For example, such a multi-node PCH may employ the redundant capabilities to support dynamic node healing under which the MN-PCH can be dynamically reconfigured such that a failed or failing primary interface or component is replaced with a corresponding spare interface or component.”)
Also, Reed further teaches,
wherein each modular component may be removed and replaced, if failing, without affecting a corresponding non-failing modular component.( provisional par 97 “Dynamic reconfiguration in the Riptide design or implementation has an additional benefit for resiliency. Physical devices may be taken out of service for repair, replacement, or upgrades, while the TidalPod and its guest operating system continues to run. Replaceable devices can be removed from the TidalPod without disrupting it.”)
Regarding claim 52, Swanson, Tati, and Reed teaches,
The method of claim 37,
Swanson further teaches,
wherein each CPU node(fig 4a:Node1, Node2; par 48 “In the illustrated embodiment of FIG. 4a, system architecture 400 includes a pair of nodes labeled Node1 and Node2, as before. For illustrative purposes, Node2 is depicted as a simple block, while Node1 is shown in further detail;”) and each IO domain are a modular component(fig 2:”HSIO N1”,”HSIO N2”,”HSIO N3”, “HSIO N4”; par 31 “In a similar manner, HSIO controllers 214 for Node 3 and Node 4 of MN-PCH 102 have been reconfigured as spare HSIO controllers 214s for each of Nadel and Node2 for MN-PCH 102a.”), wherein each modular component may be replaced, if failing, without affecting a corresponding non-failing modular component. (par 24 “In one embodiment, MN-PCH 102 supports dedicated platform level signals per node for per-node power delivery control. As a result, individual nodes can execute state transitions and host partition resets without impacting other nodes.”; par 29 “In some embodiments, a multi-node PCH may be reconfigured to support redundant capabilities to increase reliability. For example, such a multi-node PCH may employ the redundant capabilities to support dynamic node healing under which the MN-PCH can be dynamically reconfigured such that a failed or failing primary interface or component is replaced with a corresponding spare interface or component.”)
Also, Reed further teaches,
wherein each modular component may be removed and replaced, if failing, without affecting a corresponding non-failing modular component. ( provisional par 97 “Dynamic reconfiguration in the Riptide design or implementation has an additional benefit for resiliency. Physical devices may be taken out of service for repair, replacement, or upgrades, while the TidalPod and its guest operating system continues to run. Replaceable devices can be removed from the TidalPod without disrupting it.”)
Regarding claim 53, Swanson teaches,
A method of performing input/output (IO) domain failover in a fault tolerant computer system (fig 1a: node1, node2; par 32 “Generally, reconfiguration of various components and interfaces on a multi-node PCH may be implemented via switching circuitry and control logic embedded on the PCH or a different chipset interconnect from the multi-node PCH to a CPU I/O interface. Moreover, the multi-node PCH chipset, with System Management Mode (SMM) support, can dynamically change the links while preserving system context and not requiring an Operating System (OS) reset. Additionally, support for this functionality does not require any OS changes and provides self-healing for any component in the MN-PCH required for standard OS survivability and support. Also, it provides resilience for other non-critical components in the MN-PCH, include both high-speed and legacy interfaces and controllers.”); comprising:
providing the fault tolerant system, the fault tolerant system comprising having a plurality of physical CPU nodes and a plurality of physical IO domains; (fig 1a: node1, node2; par 32 “Generally, reconfiguration of various components and interfaces on a multi-node PCH may be implemented via switching circuitry and control logic embedded on the PCH or a different chipset interconnect from the multi-node PCH to a CPU I/O interface. Moreover, the multi-node PCH chipset, with System Management Mode (SMM) support, can dynamically change the links while preserving system context and not requiring an Operating System (OS) reset. Additionally, support for this functionality does not require any OS changes and provides self-healing for any component in the MN-PCH required for standard OS survivability and support. Also, it provides resilience for other non-critical components in the MN-PCH, include both high-speed and legacy interfaces and controllers.”)
designating one of the CPU nodes a standby CPU node and designating one or more of the other CPU nodes as active CPU nodes(fig 1a; par 30 “For example, in one embodiment, MNPCH 102 and MN-PCH 102a include the same components, interconnects, and logic, while MN-PCH 102 is configured to support four nodes and MN-PCH 102a is configured to support two nodes under which the (now) unused components and interconnects for the removed nodes (Node3 and Node 4) are used as spares configured to replace primary components and interconnects via associated failover operations.”), wherein each CPU node comprises a processor and a memory, wherein one or more of the active CPU nodes is executing an operating system and one or more customer applications; (fig 4:408,406,”operating system”, “software applications”; par 49 “Each of Node1 and Node2 include a processor 404 and local system memory 406.”; par 52 “Subsequently, software components comprising the operating system instance will be loaded into a protected region of memory 404. After the OS has booted, software components used for software applications may be loaded into a user space region of memory 406 allocated for applications by the OS.”)
designating at least one of the active IO domains as a primary IO domain; (fig 2a:214,214S; par 31 “In a similar manner, HSIO controllers 214 for Node 3 and Node 4 of MN-PCH 102 have been reconfigured as spare HSIO controllers 214s for each of Node1 and Node2 for MN-PCH 102a.”);
connecting each CPU node to each IO domain using a switching fabric,(par 135 “a plurality of sets of high-speed Input-Output (IO) controllers, coupled to the high-speed root fabric, each configurable to be dedicated for use by a respective node, …”)
enabling a failure trigger for each switching fabric control component in each IO domain to detect a failure of the primary or secondary IO domain or a component thereof, (fig 3a:306; par 36 “In response to the asserted error, a determination is made in a decision block 306 to whether the error is fatal or otherwise uncorrectable.”)
upon failure trigger occurring, stopping drivers using the failing IO domain or failing IO component; (fig 3c; par 44 “In a decision block 354 a determination is made to whether the failover is a controller failover. If the answer is YES, the flow proceeds to a block 354 in which the failing controller is disabled and the failover controller (e.g., the spare controller) is enabled. The interconnect hierarchy is then reconfigured in a block 356, resulting in the chipset being healed, as depicted in a block 358.”)
continuing to use the remaining operational IO domains and IO components; (fig 3c; par 44 “In a decision block 354 a determination is made to whether the failover is a controller failover. If the answer is YES, the flow proceeds to a block 354 in which the failing controller is disabled and the failover controller (e.g., the spare controller) is enabled. The interconnect hierarchy is then reconfigured in a block 356, resulting in the chipset being healed, as depicted in a block 358.”)
if one of: a failure, a beginning of a failure and a predicted failure occurs in an active node(fig 3a:304,306; par 35 “In a block 304 an error is detected and asserted. There are various well-known mechanisms for asserting errors on platforms,…. Similarly, there are well-known mechanisms for detecting errors.” par 45 “In a block 364 a determination is made to whether the node or a node controller has failing or has failed. If the node or controller is detected as failing/failed, the answer to decision block 364 is YES, and the flow proceeds to a block 366 in which the failing/failed controller is disabled and the failover controller is enabled.”), transferring current processor state of the active CPU node to the standby CPU node; (par 34 “When SMM is invoked through an SMI, the processor (e.g., Node CPU) saves its current state and switches to a separate operating environment … When the SMI handler has completed its operations, it executes a resume instruction. This instruction causes the processor to reload its saved state, switch back to protected or real mode, and resume executing the interrupted application or OS tasks.” Once the failing CPU Node has saved its operating state, the failing node is disabled and replaced with the failover node(explained in fig 3a:366; par 45 “the flow proceeds to a block 366 in which the failing/failed controller is disabled and the failover controller is enabled.”), which becomes the new active node, then the healed(new active node) processor reloads the failing node’s saved state and continues the interrupted application and operating system tasks.)
and continue executing, using the new active CPU node, the transferred operating system and the one or more customer applications,(par 45 “In a block 364 a determination is made to whether the node or a node controller has failing or has failed. If the node or controller is detected as failing/failed, the answer to decision block 364 is YES, and the flow proceeds to a block 366 in which the failing/failed controller is disabled and the failover controller is enabled.”, par 46 “The flow next proceeds from either block 366 or 368 to a block 370 in which the internal decode path is reconfigured. Following this, the DMI link is returned to a normal power and operating state”. Once the failing CPU Node has saved its operating state, the failing node is disabled and replaced with the failover node, which becomes the new active node, then the healed(new active node) processor reloads the failing node’s saved state and continues the interrupted application and operating system tasks.)
wherein each CPU node (fig 4a:Node1, Node2; par 48 “In the illustrated embodiment of FIG. 4a, system architecture 400 includes a pair of nodes labeled Node1 and Node2, as before. For illustrative purposes, Node2 is depicted as a simple block, while Node1 is shown in further detail;”) and each IO domain are a modular component(fig 2:”HSIO N1”,”HSIO N2”,”HSIO N3”, “HSIO N4”; par 31 “In a similar manner, HSIO controllers 214 for Node 3 and Node 4 of MN-PCH 102 have been reconfigured as spare HSIO controllers 214s for each of Nadel and Node2 for MN-PCH 102a.”), wherein each modular component may be replaced, if failing, without affecting a corresponding non-failing physical modular component and one or more customer applications executing thereon. (par 24 “In one embodiment, MN-PCH 102 supports dedicated platform level signals per node for per-node power delivery control. As a result, individual nodes can execute state transitions and host partition resets without impacting other nodes.”; par 29 “In some embodiments, a multi-node PCH may be reconfigured to support redundant capabilities to increase reliability. For example, such a multi-node PCH may employ the redundant capabilities to support dynamic node healing under which the MN-PCH can be dynamically reconfigured such that a failed or failing primary interface or component is replaced with a corresponding spare interface or component.”)
However, although Swanson teaches saving the current processor state of the failing active CPU node and resuming from the saved state on the standby node, Swanson does not specifically teach transferring memory contents of the active CPU node to the standby CPU node.
On the other hand, Tati teaches
A method of providing a fault tolerant computer system having a plurality of physical CPU nodes; the method comprising: (Par 9 “Techniques for implementing application fault tolerance via battery-backed replication of volatile state are provided. In one set of embodiments, a primary host system can detect a failure that causes an application of the primary host system to stop running. In response to detecting the failure, the primary host system can replicate volatile state that is used by the application to a secondary host system, where the secondary host system maintains a copy of the application, and where execution of the application is failed over to the copy on the secondary host system using the replicated volatile state.”; fig 2; par 27)
designating one of the CPU nodes a standby CPU node and designating one or more of the other CPU nodes as active CPU nodes, (fig 2:104”primary host system”, “secondary host system”; par 27 “As shown in FIG. 2, a novel post-fail agent 204 implemented in OS 108 or the BIOS of primary host system 102 can, upon the occurrence of a host failure, replicate the contents of application volatile state 120 to a corresponding agent 206 on secondary host system 104.”) wherein each CPU node comprises a processor and a memory, wherein one or more of the active CPU nodes is executing an operating system and one or more customer applications;(fig 2:110,108,122; par 21 “As shown, architecture 100 includes a primary host system 102, a secondary host system 104, …. Primary host system 102 includes an OS 108 and an FT-protected application 110 …. Secondary host system 104 includes an OS 112 and a copy, or replica, 114 of application 110. OSs 108 and 112 can be conventional "bare-metal" OSs ( e.g., Linux, Windows, etc.) or virtualization hypervisors (e.g., ESXi, Hyper-V, etc.). Similarly, application 110 can be a conventional software application or a virtual machine (VM).; par 22 “… FT managers 116 and 118 can coordinate to migrate the volatile state of application 110 (shown via reference numeral 120 in volatile memory 122) from primary host system 102 to secondary host system 104, thereby creating an initial instance of secondary copy 114 and corresponding volatile state 124 in volatile memory 126.”; par 25)
if one of: a failure, a beginning of a failure and a predicted failure occurs in an active node, (fig 3:302; par 36 “Starting with block 302, primary host system 102 can detect the occurrence of a failure that prevents application 110 from continuing its execution. Examples of such failures include, e.g., a loss of AC power, an OS-level crash, a "hard" crash caused by, e.g., an uncorrectable memory error, and a system shutdown initiated by a hard press of the system's power button.”) transferring processor state and memory contents of the active CPU node to the standby CPU node; (fig 3:310; par 38 “Upon being invoked, post-fail agent 204 can take a checkpoint of volatile state 120 of application 110 and copy the entire contents of this state (as captured in the check point) to agent 206 of secondary host system 104 (block 310). Volatile state 120 can include, e.g., the volatile memory allocated to application 110, as well as other non-persistent information pertaining to application 110 (e.g., virtual device state if application 110 is a VM).”)
and continue executing, using the new active CPU node, the transferred operating system and the one or more customer applications, (fig 3:314; par 40 “Finally, once the entirety of volatile state 120 has been successfully copied to secondary host system 104, post-fail agent 204 can send a signal to agent 206 indicating that the application can be failed over to secondary host system 104 (block 314) and workflow 300 can end.”)
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify Swanson to incorporate the transferring memory contents of the active CPU node to the standby CPU node of Tati. One of ordinary skill in the art would have been motivated to remedy the shortcomings of Swanson -- a need for a solution for the issue of how to smoothly resume operations on the failover host(Tati par 7 “A significant advantage of this approach is that it has no impact on the application's performance during normal operation. However, the application's volatile state on the original host system is lost during the fail-over and is recreated on the new host system; thus, any non-committed transactions at the point of failure are aborted. Further, the fail-over time for HA (which requires restarting the application on the new host system) is not instantaneous and can potentially take several seconds or minutes.”) -- with Tati providing a known method to solve a similar problem. Tati provides “Techniques for implementing application fault tolerance via battery-backed replication of volatile state are provided. In one set of embodiments, a primary host system can detect a failure that causes an application of the primary host system to stop running. In response to detecting the failure, the primary host system can replicate volatile state that is used by the application to a secondary host system, where the secondary host system maintains a copy of the application, and where execution of the application is failed over to the copy on the secondary host system using the replicated volatile state.”(Tati par 9)
However, Swanson and Tati do not specifically teach wherein each modular component may be physically removed and replaced.
On the other hand, Reed teaches,
A method of performing dynamic reconfiguration in a fault tolerant computer system(provisional par 250 “Described above are example details regarding Riptide, which divides the TS hyper-kernel into two layers, one dealing with implementation of a single scalable virtual Intel hardware system, and a lower layer that provides abstractions of Logical Modules used to implement all of the functions needed in a way that allows for dynamic reconfiguration while TidalPod continues to run.”; ) comprising:
providing the fault tolerant system, the fault tolerant system comprising having a plurality of physical CPU nodes and a plurality of physical IO domains;( provisional par 100 “Various examples of modules in a TidalPod will be described in further detail below. Some examples include: a node ( one or more processor chips, a motherboard, its power supply, and a number of memory DIMMs, etc.), the interconnect network (the switches, cables, and NI Cs that provide inter-node connectivity),”; provisional par 242 “Riptide's framework achieves this by using multiple ports on each node and multiple switches, such that the failure of a switch or a cable merely degrades performance, but an alternate delivery path always exists”; Switches and cables are physical components represented by a logical module; provisional par 132 ” In some embodiments, a BusPort is the logical module that represents a highspeed interconnection from a Node to the other logical nodes attached to an Internet switch. In some embodiments, there is one held in each operational node.”; provisional par 245,246,247)
designating one of the CPU nodes a standby CPU node and designating one or more of the other CPU nodes as active CPU nodes, wherein each CPU node comprises a processor and a memory, ( provisional par 80” Considering, for example, that since resources are all mobile, previously configured hot standby machines can be utilized. When failure is suspected to occur in the near future due, for example, to soft ECC errors, server temperature rising, or higher than normal network anomalies, it may be dealt with, for example, by dynamically adding an additional hot standby node to the cluster, informing all nodes about the pending node failure so they do not migrate any virtual processors to it, and having the failing node evict virtual processors at the earliest possible time, and pages of memory in active use.”) wherein one or more of the active CPU nodes is executing an operating system and one or more customer applications; (provisional par 86 “As used herein, dynamic reconfiguration refers to the capability of changing the set of hardware components contained in a TidalPod while the system continues to run a guest operating system and its applications.”)
enabling a failure trigger for each component in each domain to detect a failure of the of the component thereof, upon failure trigger occurring, stopping drivers using the failing component,(provisional par 77 “a) early indicators signal a possible impending hardware failure,”; provisional par 80 “Considering, for example, that since resources are all mobile, previously configured hot standby machines can be utilized. When failure is suspected to occur in the near future due, for example, to soft ECC errors, server temperature rising, or higher than normal network anomalies, it may be dealt with, for example, by dynamically adding an additional hot standby node to the cluster, informing all nodes about the pending node failure so they do not migrate any virtual processors to it, and having the failing node evict virtual processors at the earliest possible time, and pages of memory in active use.”; provisional par 426, par 427)
continuing to use the remaining operational IO domains and IO components; ( provisional par 242 “In order to make the Interconnect among nodes resilient, redundancy is added. In some embodiments, Riptide's framework achieves this by using multiple ports on each node and multiple switches, such that the failure of a switch or a cable merely degrades performance, but an alternate delivery path always exists.”; provisional par 132(defines BusPort); provisional par 245,246,247(describes the logical node failover options))
if one of: a failure, a beginning of a failure and a predicted failure occurs in an active node, transferring processor state and memory contents of the active CPU node to the standby CPU node; (provisional par 242 “Riptide's framework achieves this by using multiple ports on each node and multiple switches, such that the failure of a switch or a cable merely degrades performance, but an alternate delivery path always exists.”; provisional par 243 “This implies that selection of paths and detection of failed paths is done in the network kernel driver.”; provisional par 77 “a) early indicators signal a possible impending hardware failure,”; provisional par 80 “Considering, for example, that since resources are all mobile, previously configured hot standby machines can be utilized. When failure is suspected to occur in the near future due, for example, to soft ECC errors, server temperature rising, or higher than normal network anomalies, it may be dealt with, for example, by dynamically adding an additional hot standby node to the cluster, informing all nodes about the pending node failure so they do not migrate any virtual processors to it, and having the failing node evict virtual processors at the earliest possible time, and pages of memory in active use.”; provisional par 426, par 427)
and continue executing, using the new active CPU node, the transferred operating system and the one or more customer applications, (provisional par 86 “As used herein, dynamic reconfiguration refers to the capability of changing the set of hardware components contained in a TidalPod while the system continues to run a guest operating system and its applications.”; provisional par 97 “Dynamic reconfiguration in the Riptide design or implementation has an additional benefit for resiliency. Physical devices may be taken out of service for repair, replacement, or upgrades, while the TidalPod and its guest operating system continues to run. Replaceable devices can be removed from the TidalPod without disrupting it.”; provisional par 972 “From the guest operating system's point of view, it did not observe a fault (the hyper-kernel intercepted the faults and performed the vm exit/enter). Instead, the guest operating system attempted to access a page, and at the next instruction, it has already accessed the page (where the guest operating system is not aware of the underlying migration that was performed by the hyper-kernel).”;)
wherein each CPU node and each IO domain are a modular component, wherein each modular component may be physically removed and replaced, if failing,(provisional par 86 “As used herein, dynamic reconfiguration refers to the capability of changing the set of hardware components contained in a TidalPod while the system continues to run a guest operating system and its applications.”; provisional par 80; provisional par 250; provisional par 97 “Dynamic reconfiguration in the Riptide design or implementation has an additional benefit for resiliency. Physical devices may be taken out of service for repair, replacement, or upgrades, while the TidalPod and its guest operating system continues to run. Replaceable devices can be removed from the TidalPod without disrupting it. Moreover, they may remain in place for an arbitrary period until a removal for replacement or upgrade can be scheduled. In many datacenter environments, such flexibility can be quite valuable - one both reduces downtime by continuing to run, and at the same time allows the repair to be done right, without need for "emergency on-call specialists".) without affecting a corresponding non-failing physical modular physical component and one or more customer applications executing thereon. (provisional par 86 “As used herein, dynamic reconfiguration refers to the capability of changing the set of hardware components contained in a TidalPod while the system continues to run a guest operating system and its applications.”;)
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify Swanson and Tati to incorporate CPU node and IO domain are configured to be removed and replaced without affecting applications executing on one or more of the other CPU nodes and IO domains of Reed. One of ordinary skill in the art would have been motivated to remedy the shortcomings of Swanson and Tati -- a need for how to replace components after they have failed(Reed provisional par 91 “It may not be possible to build a system that never fails. However, years of experience have taught the IT industry that the distribution of failures among component types is typically extremely skewed. CPUs, for example, almost never fail, while network cables and switches fail quite frequently, power supplies overheat, fans fail, etc.”) -- with Reed providing a known method to solve a similar problem. Reed provides “As described herein, Riptide provides a framework for exploiting standard redundant hardware techniques that drastically reduce the impact of failure of the components that dominate the causes of downtime. Using the techniques described herein. hardware failures are transformed into degradation of performance, which can then be dealt with by dynamic reconfiguration. This type of approach provides an arbitrary number of"9's" of availability.”(Reed provisional par 91)
Regarding claim 54, Swanson, Tati, and Reed teaches,
The method of claim 24,
Swanson further teaches,
wherein the switching fabric comprises a first switch and a second switch, wherein the active IO domain comprises the first switch and a second IO domain comprises the second switch.(par 32 “Generally, reconfiguration of various components and interfaces on a multi-node PCH may be implemented via switching circuitry and control logic embedded on the PCH or a different chipset interconnect from the multi-node PCH to a CPU I/O interface. Moreover, the multi-node PCH chipset, with System Management Mode (SMM) support, can dynamically change the links while preserving system context and not requiring an Operating System (OS) reset. Additionally, support for this functionality does not require any OS changes and provides self-healing for any component in the MN-PCH required for standard OS survivability and support. Also, it provides resilience for other non-critical components in the MN-PCH, include both high-speed and legacy interfaces and controllers.”)
Also, Reed further teaches,
wherein the switching fabric comprises a first switch and a second switch, wherein the active IO domain comprises the first switch and a second IO domain comprises the second switch.( provisional par 100 “Various examples of modules in a TidalPod will be described in further detail below. Some examples include: a node (one or more processor chips, a motherboard, its power supply, and a number of memory DIMMs, etc.), the interconnect network (the switches, cables, and NICs that provide inter-node connectivity), and an addressable guest network interface ( one port on hardware NIC).”; provisional par 132 “In some embodiments, a BusPort is the logical module that represents a highspeed interconnection from a Node to the other logical nodes attached to an Internet switch. In some embodiments, there is one held in each operational node.”; provisional par 242 “In order to make the Interconnect among nodes resilient, redundancy is added. In some embodiments, Riptide's framework achieves this by using multiple ports on each node and multiple switches, such that the failure of a switch or a cable merely degrades performance, but an alternate delivery path always exists.”)
Regarding claim 55, Swanson, Tati, and Reed teaches,
The system of claim 37,
Swanson further teaches,
wherein each IO domain of the plurality of IO domains comprises an IO board.(fig 2a:214,214S; par 31 “In a similar manner, HSIO controllers 214 for Node 3 and Node 4 of MN-PCH 102 have been reconfigured as spare HSIO controllers 214s for each of Node1 and Node2 for MN-PCH 102a.”);
Regarding claim 57, see the teachings of Swanson, Tati, and Reed with respect to claim 54 above. Claim 57 is rejected for the same reasons as claim 54.
Claim(s) 26,41 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20150370661 A1 (Swanson), US 20190129814 A1 (Tati), and US 20190149399 A1 (Reed) as applied to claims 24 and 37 above, and further in view of US 20170235950 A1 (Gopalapura Venkatesh, herein referred to as Venkatesh).
Regarding claim 26, Swanson, Tati, and Reed teaches,
The method of claim 24,
Swanson further teaches,
further comprising running a provisioning service for each active IO domain (par 28 “The manageability engines (ME/CSME/IE) likewise support dedicated interfaces for each node, while providing access to manageability sensors and IO's on a per-node basis.”).
However, Swanson, Tati, and Reed do not specifically teach wherein each provisioning service communicates with the provisioning service of the other active IO domain to form a unified hierarchy of physical and/or virtual functions.
On the other hand, Venkatesh teaches
A self-healing virtualized file server (par 8 “In particular embodiments, a virtualized file server (VFS) self-healing system may automatically identify data corruption and perform data recovery operations at multiple levels in the storage hierarchy, including the file level, filesystem level, and storage level.”)
further comprising running a provisioning service for each active IO domain, wherein each provisioning service communicates with the provisioning service of the other active IO domain(fig 2A:170a-c; par 265 “The cluster health service or the like, e.g., FSVMs 170 monitoring each other, may detect communication timeouts, I/O alerts, or other events that potentially indicate a host machine 201 has failed.”) to form a unified hierarchy of physical and/or virtual functions.(par 48 “CVMs 110a-c are used to manage storage and input/output ("I/O") activities according to particular embodiments. These special VMs act as the storage controller in the currently described architecture. Multiple such storage controllers may coordinate within a cluster to form a unified storage controller system.”)
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify Swanson, Tati, and Reed to incorporate the distributed provisioning service of Venkatesh. One of ordinary skill in the art would have been motivated to remedy the shortcomings of Swanson, Tati, and Reed -- a need for a solution for the issue of how to handle failover if the leader is lost-- with Venkatesh providing a known method to solve a similar problem. Venkatesh provides “In particular embodiments, since VFS compute and storage units may be distributed across multiple FSVMs on multiple host machines, the self-healing system may efficiently monitor the corruption and data loss in a parallel and distributed fashion on all the FSVMs or host machines of the VFS and detect and recover the particular data on the corresponding FSVM or host machine on which the data is located.” (Venkatesh par 10)
Regarding claim 41, it is the system implementing the method of claim 26 and is rejected for the same reasons.
Claim(s) 34 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20150370661 A1 (Swanson), US 20190129814 A1 (Tati), and US 20190149399 A1 (Reed) as applied to claim 24 above, and further in view of US 20170286352 A1 (Kumar).
Regarding claim 34, Swanson, Tati, and Reed teaches,
The method of claim 24
Swanson further teaches,
further comprising running one or more management engine instances using a management processor in each IO domain, wherein each management engine queries the switching fabric control components connected to the respective management engine to obtain an enumerated hierarchy (par 162 “FIG. 11 illustrates an example method 1100 for ingesting data into a virtualized file server 1208. The method 1100 may be performed primarily by, for example, one of the host machines 201. The method 1100 begins at step 1102 by connecting to an existing virtualized file server (VFS) 1202. Step 1104 may retrieve a list of existing storage items 1204, e.g., shares, directories, and/or files, that are located on the existing VFS 1202, and their sizes. The list may be retrieved recursively to identify nested directories.”) of physical and/or virtual functions on a per control component basis (par 36 “More particularly, the component failure counters are implemented for components or aggregations of components that support failover. Such components generally may include interfaces, controllers, and components serving dedicated functions, wherein a spare or failover instance of the component is available as a replacement.”), wherein each management engine stores enumerated per-component hierarchies into a per-domain hierarchy of physical and/or virtual functions within the IO domain associated with each management engine.(par 76 “FIG. 3A illustrates an example hierarchical structure 300 of a VFS instance in a cluster according to particular embodiments.”)
However, Swanson, Tati, and Reed do not specifically teach wherein each management engine merges enumerated per-component hierarchies into a per-domain hierarchy of physical and/or virtual functions
On the other hand, Kumar teaches
A method of PCIe cable topology discovery in a data center environment (Par 24 “Mechanisms for PCie cable topology discovery in a Rack Scale Architecture environment and associated methods, apparatus, and systems are described herein.”),
further comprising running one or more management engine instances (fig 2:206; par 29 “In some embodiments, groups of computing racks 202 are managed as separate pods via pod manager(s) 206. In one embodiment, a single pod manager is used to manage all of the racks in the pod. Alternatively, distributed pod managers may be used for pod management operations.”) using a management processor in each IO domain(fig 4:408; par 33 “This role is fulfilled by Rack Management Module (RMM), along with a sub-rack unit (the drawer units in RSA terminology) manager called a Pooled System Management Engine (PSME)”), wherein each management engine queries the switching fabric control components connected to the respective management engine to obtain an enumerated hierarchy of physical and/or virtual functions on a per control component basis(par 54 “In a block 1004, the POD manager instructs the PSME on each RSA drawer to report the UID of the emulated device(s) as seen by each of its server nodes.”), wherein each management engine merges enumerated per-component hierarchies into a per-domain hierarchy of physical and/or virtual functions within the IO domain associated with each management engine.(par 54 “On the basis of the UIDs read by the RSA drawer PSMEs, the POD manager constructs the topology of the PCie cabling (i.e., which PNC cable bundle is attached to which RSA drawer) and communicates the topology to the PNCs, as depicted in a block 1006.”)
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify Swanson, Tati, and Reed to incorporate the topology discovery of Kumar. One of ordinary skill in the art would have been motivated to remedy the shortcomings of Swanson, Tati, and Reed -- a need for a solution for the issue of how to more efficiently manage resources of data center racks at the component level (Kumar par 5 “Recently, Intel® Corporation introduced new rack architecture called Rack Scale Architecture (RSA). Rack Scale Architecture is a logical architecture that disaggregates compute, storage, and network resources and introduces the ability to pool these resources for more efficient utilization of assets.”) -- with Kumar providing a known method to solve a similar problem. Kumar provides “Mechanisms for PCie cable topology discovery in a Rack Scale Architecture environment and associated methods, apparatus, and systems are described herein.” (Kumar par 24)
Claim(s) 36 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20150370661 A1 (Swanson), US 20190129814 A1 (Tati), and US 20190149399 A1 (Reed) as applied to claims 24 above, and further in view of US 20180349231 A1 (Panda).
Regarding claim 36, Swanson, Tati, and Reed teaches,
The method of claim 24
Swanson further teaches,
further comprising the ability to bring the failing IO domain or the failing IO component back into service by asserting an independent reset to the failing IO domain or failing IO component. (par 24 “As a result, individual nodes can execute state transitions and host partition resets without impacting other nodes. The reset sequencing flows for the individual nodes are mutually independent, and can be interleaved by PMC208.”)
However, although Swanson teaches an independent reset to nodes, Swanson, Tati, and Reed does not specifically anticipate triggering this reset attempting to bring the failing IO domain or the failing IO component back into service.
On the other hand, Panda teaches
A method for delayed error processing (par 34 “A system and method for delayed error processing will now be described ….”)
further comprising attempting to bring the failing IO domain or the failing IO component back into service by asserting an independent reset to the failing IO domain or failing IO component. (par 23 “The ultimate response to an uncorrectable error may depend on system capabilities. For example, the response could include a shutdown and restart or error recovery. In some cases, where error recovery is disabled or not available, the response may generally be a shutdown of the compute resource, followed by error harvesting, and then a restart.”)
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to further modify Swanson, Tati, and Reed to incorporate the error response of Panda. One of ordinary skill in the art would have been motivated to remedy the shortcomings of Swanson, Tati, and Reed -- a need for a solution for the issue of how to respond to errors (Panda par 16 “One challenge in such a multi-containerized system is that a single uncorrectable error in one container can cause a failure of the hierarchical structures and in some cases may bring down the underlying operating system, thus causing data loss across other containers.” Par 20 “It is thus advantageous to provide a system that can more gracefully recover from uncorrectable errors.”) -- with Panda providing a known method to solve a similar problem. Panda provides “Embodiments of the present specification support recovery from critical errors such as memory errors and cache errors” (Panda par 27)
Response to Arguments
Applicant’s arguments, see remarks 9, filed 03/03/2026, with respect to the objection of claims 54 and 57 have been fully considered and are persuasive. The objections to claims 54 and 57 of 12/03/2025 has been withdrawn.
Applicant's arguments filed 03/03/2026 , with respect to the rejections of the independent claims under 35 U.S.C. 103 as being unpatentable over US 20150370661 A1 (Swanson) in view of US 20190129814 A1 (Tati) and US 20190149399 A1 (Reed) have been fully considered but they are not persuasive.
With respect to the independent claims, the applicant has argued that the cited part of Reed’s pre-grant publication is not entitled to the benefit of Reed’s provisional application. The examiner respectfully disagrees, but has updated the rejections with citations that come purely from the provisional application, so there is now no argument about if there is support from the provisional application or not, as all the citations of the Reed reference now come from the provisional application.
With respect to the independent claims, the applicant has further argued that Reed does not relate to a fault tolerant system and the associated features claimed, explaining that Reed relates to managing a data center and spinning up new resources or shifting resources if there is a problem with an existing processor. Further explaining that if Reed’s management software fails, then faults will not be properly managed. The examiner respectfully disagrees. Reed explicitly addresses system resiliency and continuity of operation in the presence of hardware failures, which are core aspects of fault tolerant systems. Reed teaches this in the cited (provisional par 86 “As used herein, dynamic reconfiguration refers to the capability of changing the set of hardware components contained in a TidalPod while the system continues to run a guest operating system and its applications.”; provisional par 97 “Dynamic reconfiguration in the Riptide design or implementation has an additional benefit for resiliency. Physical devices may be taken out of service for repair, replacement, or upgrades, while the TidalPod and its guest operating system continues to run. Replaceable devices can be removed from the TidalPod without disrupting it.”;). Reed also mentions “fault tolerance” specifically in (provisional par 251 “In some embodiments, Logical Modules encapsulate the mechanisms of fault tolerance by supporting operation in a "degraded" state when a hardware component used in their implementation fails.”). In response to applicant's argument that Reed is nonanalogous art, it has been held that a prior art reference must either be in the field of the inventor’s endeavor or, if not, then be reasonably pertinent to the particular problem with which the inventor was concerned, in order to be relied upon as a basis for rejection of the claimed invention. See [MPEP 2141.01(a) “Analogous and Nonanalogous Art”]. In this case, although Reed may not use the exact terminology “fault tolerant system”, the disclosed functionality clearly addresses the same technical problem and provides the same type of solution as fault tolerant systems.
With respect to the independent claims, the applicant has further argued that Reed provides no disclosure about a standby node taking over for an active node, specifically covered in limitations “wherein if one of: the failure, a beginning of the failure and the predicted failure occurs in an active node, processor state and memory contents of the active CPU node is transferred to the standby CPU node; wherein the memory contents and processor state comprise the state of the running operating system and the one or more customer applications such that the operating system and the one or more customer applications are transferred to the standby CPU node, wherein the standby CPU node becomes new active CPU node and continues executing the transferred operating system and the one or more customer applications.”. The examiner respectfully disagrees. Reed teaches creating a continuation object that moves a computation from a source node to a target node, copying all of its contents from the source to the target in the cited (provisional par 428 “Continuations are built using the "InitContinuation" routine. If a decision is made to move the computation, the remote physical node holding the resource will build a continuation that corresponds to the stalled computation and will store it in the remote physical node's event table. When that continuation resumes, the resource will be available. In effect, the hyperkernel has transferred the virtual processor to a different node.”; provisional par 513 “In the event an addresses needs to be put into a continuation, care is taken in the move, since the address is a physical address of the source, and bears no relationship with the physical address in the destination. Moving a continuation means copying its contents to the destination node as discussed above, and remapping any physical addresses from the source to the target.”). Reed also explains that this transition allows the operating system and its applications to continue to run normally in the cited (provisional par 86 “As used herein, dynamic reconfiguration refers to the capability of changing the set of hardware components contained in a TidalPod while the system continues to run a guest operating system and its applications.”). The examiner interprets this transfer of processing from one node to a different node while allowing the guest operating system and it’s applications to run normally as limitations “processor state and memory contents of the active CPU node is transferred to the standby CPU node; wherein the memory contents and processor state comprise the state of the running operating system and the one or more customer applications such that the operating system and the one or more customer applications are transferred to the standby CPU node, wherein the standby CPU node becomes new active CPU node and continues executing the transferred operating system and the one or more customer applications.”.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20190356609 A1 - Grunwald - switching between mediator services for a storage system.
US 20160077937 A1 - Inforzato - from parent app international search report. Uses a switching fabric to connect compute nodes to IO nodes.
US 10613914 B2 - Carlen - container domain orchestration service.
US 20150370666 A1 - Breakstone - cited in ISR, has processors connected through PCIe to storage drives. Processors fail over to another processor. a really good reference too.
US 20130058351 A1 - Casado - cited in ISR, has failover and re-directs packets to other locations.
US 20230393956 A1 - Guim Bernat - transfers network device state during failover. Published after provisional date.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL XU whose telephone number is (571)272-5688. The examiner can normally be reached Monday-Friday 8:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bryce Bonzo can be reached at (571) 272-3655. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL XU/Examiner, Art Unit 2113
Prosecution Insights