DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 21-40 are pending in this application.
Response to Arguments
Applicant’s arguments regarding the rejections of claims 21-40 under 35 U.S.C. 112b have been fully considered and are persuasive. The rejections have been withdrawn. However, new 35 U.S.C. 112b rejections are applied to claims 32-40 based on the amendments.
Applicant's arguments regarding the 35 U.S.C. 102/103 rejections of claims 21-40 have been fully considered but they are not persuasive.
Regarding the 35 U.S.C. 102/103 rejections, the applicant argues the following in the remarks:
Sakashita fails to teach amended claim 21.
Amended claims 32 and 36 recite similar limitations to amended claim 21, so they are allowable.
Sakashita in view of McCormack do not teach claim 24.
Examiner has thoroughly considered Applicant’s arguments, but respectfully finds them unpersuasive for at least the following reasons:
As to point (a), the argument is moot in light of the references being applied in the current rejection.
As to point (b), the examiner respectfully disagrees. Amended claims 32 and 36 are not similar in scope to claim 21.
As to point (c), the argument is moot in light of the references being applied in the current rejection.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 32-40 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
As per claim 32:
Lines 10-12 recite “wherein each of the work portions represents a contribution of a corresponding volume to the load” but it is unclear what the corresponding volume of each work portion refers to.
As per claim 36:
Lines 7-9 recite “wherein the work portions comprise headroom values representing used performance capacity attributable to the volumes” but it is unclear performance capacity of what component is being used.
Claims 33-35, and 37-40 are dependent claims of claims 32 and 36, and fail to resolve the deficiencies of claims 32 and 36, so they are rejected for the same reasons.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 21-31 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-19 of U.S. Patent No. 12050938 in view of Sakashita et al. (US 20210105322 A1 hereinafter Sakashita), in view of Oe et al. (US 20100235604 A1 hereinafter Oe), and further in view of Kachare et al. (US 20190272012 A1 hereinafter Kachare).
Although the claims at issue are not identical, they are not patentably distinct from each
other.
Regarding claim 21 of the instant application, the following table compares claim 21 with
claim 3 of U.S. Patent No. 12050938. The differences are bolded.
Instant application
U.S. Patent No. 12050938
21. A method for correcting load imbalances in a cluster including computing nodes, the method comprising:
identifying a node of the computing nodes having a load that is highest of loads in the cluster;
calculating work portions of volumes on the node, wherein each of the work portions represents a percentage of the load;
identifying a candidate subset of the volumes, wherein the candidate subset includes one or more of the volumes associated with a subset of the work portions representing percentages of the load below a threshold percentage of the load;
determining a move subset from the candidate subset based on a performance impact of moving respective volumes in the candidate subset from the node; and moving the move subset from the node to one or more other nodes of the computing nodes.
1. A method comprising: analyzing, by a performance manager executed by a processor, a load imposed by one or more workloads operating on each node in a plurality of nodes; selecting, by the performance manager based on the analyzing the plurality of nodes, a first node from the plurality of nodes having a highest load of the plurality of nodes; in response to selecting the first node, analyzing by the performance manager a work portion of each volume of a plurality of volumes hosted on the first node; selecting, by the performance manager based on the analyzing the plurality of volumes, a first volume from the plurality of volumes for movement to a second node based on the work portion of the first volume being greater than corresponding work portions of a remainder of the plurality of volumes on the first node; moving, by the performance manager in response to the highest load corresponding to the first node being less than an estimated optimal performance capacity of the first node and before latency for the first node increases more than a utilization increase of a resource of the first node, the first volume from the plurality of volumes to the second node that has a lower load than the highest load corresponding to the first node, wherein the estimated optimal performance capacity comprises a point that represents a maximum utilization of the resource of the first node and wherein beyond the estimated optimal performance capacity of the first node, an increase in the load on the first node causes a larger increase in latency than the utilization increase of the resource; imposing, by the performance manager in response to the highest load corresponding to the first node being equal to the estimated optimal performance capacity, a first Quality of Service (QoS) limit on at least one workload from the one or more workloads operating on the first node; and imposing, by the performance manager in response to the highest load corresponding to the first node being above the estimated optimal performance capacity, a second QoS limit on the at least one workload that comprises a rogue workload that includes an abnormal performance metric relative to historical performance data.
3. The method of claim 1, further comprising: filtering a volume of the plurality of volumes having a work portion above a threshold from being considered to be selected to be moved as part of the analyzing the work portion of each volume of the plurality of volumes hosted on the first node.
Although the claims at issue are not identical, they are not patentably distinct from each other. U.S. Patent No. 12050938 does not explicitly claim a method for correcting load imbalances in a cluster including computing nodes, the method comprising: wherein each of the work portions represents a percentage of the load; a subset of the work portions representing percentages of the load below a threshold percentage of the load; determining a move subset from the candidate subset based on a performance impact of moving respective volumes in the candidate subset from the node; and moving the move subset from the node to one or more other nodes of the computing nodes.
However, Sakashita teaches a method for correcting load imbalances in a cluster including computing nodes, the method comprising ([0001] The present invention relates to a distributed storage system and a data migration method and is preferably applied to the distributed storage system and the data migration method which are configured to migrate data between/among nodes in order to smooth a load; [0099] In step S208, the optimizer 200 assumes that the target volume which is on the migration source node which is selected in step S206 and is designated in step S207 is to be migrated to a node which is the lowest in load in all the nodes, that is, the node which is the smallest in evaluation value.): determining a move subset from the candidate subset based on a performance impact of moving respective volumes in the candidate subset from the node ([0105] Then, the optimizer 200 decides whether the goodness value which is calculated in step S213 is increased (step S214). In a case where the goodness value is increased (YES in step S214), the optimizer 200 sets the provisional volume migration to the variable “BestMove”; [0104] Further, the optimizer 200 calculates the goodness value (step S213). The goodness value is obtained by subtracting the evaluation value which is obtained after migration from the evaluation value of the entire distributed storage system 1 which is obtained before migration; [0100] Then, the optimizer 200 decides whether the capacity of the volume is smaller than a predetermined threshold value (step S209). This threshold value is defined by the user or the distributed storage system 1, for example, from the viewpoints as follows. In general, it is thought that a volume which is large in capacity becomes high also in load in proportion and therefore even in a case where the volume is migrated to another node, the possibility that the volume would bring the migration destination node into an overloaded state is high. In addition, since also the cost involved in migration is high, it is difficult to obtain advantages of migration. Accordingly, the optimizer 200 defines the predetermined threshold value. In step S209, in a case where the capacity of the volume is smaller than the predetermined threshold value (YES in step S209), the optimizer 200 proceeds to step S210. In a case where the capacity of the volume is larger than the predetermined threshold value (NO in step S209), the optimizer 200 skips processes in steps up to step S215 and decides to exit from a loop in step S207. [0101] In step S210, the optimizer 200 confirms whether the migration destination node has the free capacity which is sufficient for migration of the target volume. In a case where the sufficient free capacity is present (YES in step S210), the optimizer 200 proceeds to step S211.); and moving the move subset from the node to one or more other nodes of the computing nodes ([0099] In step S208, the optimizer 200 assumes that the target volume which is on the migration source node which is selected in step S206 and is designated in step S207 is to be migrated to a node which is the lowest in load in all the nodes, that is, the node which is the smallest in evaluation value; [0107] In step S216, in a case where “BestMove” is present (YES in step S216), the optimizer 200 adds “BestMove” concerned to “MoveCandidate” (step S217)).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the claims of U.S. Patent No. 12050938 with the teachings of Sakashita to find the best load to migrate (see Sakashita [0105] Then, the optimizer 200 decides whether the goodness value which is calculated in step S213 is increased (step S214). In a case where the goodness value is increased (YES in step S214), the optimizer 200 sets the provisional volume migration to the variable “BestMove”).
The claims of U.S. Patent No. 12050938 and Sakashita fail to teach wherein each of the work portions represents a percentage of the load; a subset of the work portions representing percentages of the load below a threshold percentage of the load.
However, Oe teaches wherein each of the work portions represents a percentage of the load ([0120] In addition, information containing "441" times/second as the number of reads (at peak), "22" times/second as the number of L0 reads (during measurement), (22/441).times.100="5%" as an L0 load (read), "45" times/second as the number of L1 reads (during measurement), (45/441).times.100="10%" as an L1 load (read), "18" times/second as the number of L2 reads (during measurement), and (18/441).times.100="4%" as an L2 load (read) is set. [0121] Furthermore, information containing "320" times/second as the number of writes (at peak), "7" times/second as the number of L0 writes (during measurement), (7/320).times.100="2%" as an L0 load (write), "20" times/second as the number of L1 writes (during measurement), (20/320).times.100="6%" as an L1 load (write), "6" times/second as the number of L2 writes (during measurement), and (6/320).times.100="2%" as an L2 load (write) is set. [0122] As a result, the L0 load (read) of "5%" and the L0 load (write) of "2%" is added up and "7%" is set to the L0 load (read+write). [0123] In addition, the L1 load (read) of "10%" and the L1 load (write) of "6%" is added up and "16%" is set to the L1 load (read+write). [0124] Furthermore, the L2 load (read) of "4%" and the L2 load (write) of "2%" is added up and "6%" is set to the L2 load (read+write). [0125] In this manner, IO size-specific read+write loads are calculated for the logical volumes 910, 920, and 930, and the read+write loads are further added up. Accordingly, breakdowns of the loads on the logical volumes 910, 920, and 930 may be evaluated; [0040] a load generated at the logical volumes A, B, and C of the storage node 2; [0065] The logical volume 910 is assigned a logical volume ID of "L0". [0066] The logical volume 920 is assigned a logical volume ID of "L1". [0067] The logical volume 930 is assigned a logical volume ID of "L2").
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined claims of U.S. Patent No. 12050938 and Sakashita with the teachings of Oe to prevent a shortage of resources (see Oe [0009] By monitoring storage area utilization, the occurrence of a shortage in a storage area may be prevented).
The claims of U.S. Patent No. 12050938, Sakashita, and Oe fail to teach a subset of the work portions representing percentages of the load below a threshold percentage of the load.
However, Kachare teaches a subset of the work portions representing percentages of the load below a threshold percentage of the load ([0100] the local service provider 50 determines (S102) whether the total power consumption of the storage bank 302 is less than a first percentage threshold (e.g., 40% or a value between 30% to 50%) of the load; [0069] if a storage device 10 changes from operating at normal 61 to operating at greater than 90%; [0019] determining whether one or more second storage devices of the plurality of storage devices are consuming power under a threshold power level; [0107] According to some embodiments, the local service provider 50 identifies (S122) which storage devices 10 consume power at a level less than a threshold power level. In some examples, the threshold may be set at 75%).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the claims of U.S. Patent No. 12050938, Sakashita, and Oe with the teachings of Kachare to promote efficiency (see Kachare [0108] The local service provider 50 then compares the actual power consumption with threshold power level to determine if consumed power of the storage device 10 is below the threshold power level. The local service provider 50 then dynamically instructs the identified storage devices 10 to operate at a power cap corresponding to the first level (e.g., at 75% or 80% of maximum power), as opposed to the default power cap of 100% maximum power. Because the power efficiency of a PSU drops as it reaches its maximum load capacity, lowering the power cap of the storage devices 10 may bring down the overall power usage of the storage bank 302, thus allowing the PSU to operate at a lower power level and at a higher (e.g., peak) power efficiency range.).
Similar claim mappings of the dependent claims of claim 21 would have been obvious to a person having ordinary skill in the art but have been omitted for the sake of brevity.
Claims 32-40 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-19 of U.S. Patent No. 12050938 in view of Sakashita et al. (US 20210105322 A1 hereinafter Sakashita).
Although the claims at issue are not identical, they are not patentably distinct from each
other.
Regarding claim 32 of the instant application, the following table compares claim 32 with
claim 8 of U.S. Patent No. 12050938. The differences are bolded.
Instant application
U.S. Patent No. 12050938
32. A computing device for correcting load imbalances in a cluster including computing nodes, the computing device comprising: a memory containing machine readable medium comprising machine executable code having stored thereon instructions for performing a method of load balancing in a storage system; and a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to: identify a node of the computing nodes having a load that is highest of loads in the cluster;
calculate work portions of volumes on the node, wherein each of the work portions represents a contribution of a corresponding volume to the load; determine a subset of the volumes to move from the node based on a performance impact of moving the work portions from the node; and move the subset of the volumes from the node to one or more other nodes of the computing nodes.
8. A non-transitory machine readable medium having stored thereon instructions for performing a method of load balancing, comprising machine executable code which when executed by at least one machine, causes the at least one machine to:
select a first node from a plurality of nodes having a highest load of the plurality of nodes;
analyze a work portion of each volume of a plurality of volumes hosted on the first node to identify a first volume from the plurality of volumes having a first work portion that is greater than corresponding work portions of a remainder of the plurality of volumes on the first node; in response to the highest load corresponding to the first node being less than an estimated optimal performance capacity of the first node, perform a preventive action that includes moving the first volume identified from the analysis to a second node that has a lower load than the highest load associated with the first node, wherein beyond the estimated optimal performance capacity of the first node, an increase in the highest load corresponding to the first node causes a larger increase in latency for the first node than a utilization increase of a resource of the first node; in response to the highest load corresponding to the first node being equal to the estimated optimal performance capacity of the first node, perform a proactive action that includes imposing a first Quality of Service (QoS) limit on one or more workloads operating on the first node; and in response to the highest load corresponding to the first node being greater than the estimated optimal performance capacity of the first node, perform a reactive action that includes imposing a second QoS limit on a rogue workload operating on the first node, the rogue workload comprising an abnormal performance characteristic relative to historical performance data.
Although the claims at issue are not identical, they are not patentably distinct from each other. U.S. Patent No. 12050938 does not explicitly claim a computing device for correcting load imbalances in a cluster including computing nodes; wherein each of the work portions represents a contribution of a corresponding volume to the load; determine a subset of the volumes to move from the node based on a performance impact of moving the work portions from the node; and move the subset of the volumes from the node to one or more other nodes of the computing nodes.
However, Sakashita teaches a computing device for correcting load imbalances in a cluster including computing nodes ([0001] The present invention relates to a distributed storage system and a data migration method and is preferably applied to the distributed storage system and the data migration method which are configured to migrate data between/among nodes in order to smooth a load; [0099] In step S208, the optimizer 200 assumes that the target volume which is on the migration source node which is selected in step S206 and is designated in step S207 is to be migrated to a node which is the lowest in load in all the nodes, that is, the node which is the smallest in evaluation value); wherein each of the work portions represents a contribution of a corresponding volume to the load (Fig. 8; [0092] the optimizer 200 rearranges the nodes and the volumes in descending order of the evaluation values in step S202 and thereby the node and the volume which are the highest in evaluation value, in other words, the node and the volume which are the highest in current load are rearranged to high-order positions; [0097] In step S206, the optimizer 200 selects the “nodeIndex”-th node which is obtained after rearrangement of the nodes and volumes in step S202 as the migration source node; [0090] Next, the optimizer 200 executes an evaluation value calculation for calculating evaluation values of the node and the volume and rearranges the nodes and the volumes in descending order of evaluation values on the basis of a calculation result of the evaluation value calculation (step S202). Then, the optimizer 200 executes processes in step S203 and succeeding steps with arrangement of the nodes and the volumes which are rearranged in step S202 being set as a standard; [0062] The migration operator 400 observes a per-volume time-series load; [0068] the node configuration table 121 has data items of the storage node ID 1211, a processor frequency 1212, the number of processors 1213, a memory 1214, a bandwidth of internode network 1215, a bandwidth of compute network 1216, the number of drives 1217, a total capacity 1218 and an used capacity 1219 and the specification of the hardware which is loaded on each node 10 is described in the table 121. In these items, a total value of the capacities of the drives which are loaded on the target node 10 is described in the total capacity 1218 and this capacity corresponds to the capacity of the drive pool 22. In addition, the total capacity which is allocated to each volume 24 from the physical drive 21 is described in the used capacity 1219; [0069] FIG. 8 is a diagram illustrating one configuration example of the volume placement table. The volume placement table 122 is a data table which indicates such volume placement that each volume 24 is to be placed on which node 10. In the example in FIG. 8, the volume arrangement table 122 has data items of a volume ID 1221, the used capacity 1222 and the storage node ID 1223. The storage node ID 1223 in FIG. 8 corresponds to the storage node ID 1211 on the node configuration table 121 in FIG. 7.); determine a subset of the volumes to move from the node based on a performance impact of moving the work portions from the node ([0100] Then, the optimizer 200 decides whether the capacity of the volume is smaller than a predetermined threshold value (step S209). This threshold value is defined by the user or the distributed storage system 1, for example, from the viewpoints as follows. In general, it is thought that a volume which is large in capacity becomes high also in load in proportion and therefore even in a case where the volume is migrated to another node, the possibility that the volume would bring the migration destination node into an overloaded state is high. In addition, since also the cost involved in migration is high, it is difficult to obtain advantages of migration. Accordingly, the optimizer 200 defines the predetermined threshold value. In step S209, in a case where the capacity of the volume is smaller than the predetermined threshold value (YES in step S209), the optimizer 200 proceeds to step S210. In a case where the capacity of the volume is larger than the predetermined threshold value (NO in step S209), the optimizer 200 skips processes in steps up to step S215 and decides to exit from a loop in step S207. [0101] In step S210, the optimizer 200 confirms whether the migration destination node has the free capacity which is sufficient for migration of the target volume. In a case where the sufficient free capacity is present (YES in step S210), the optimizer 200 proceeds to step S211.); and move the subset of the volumes from the node to one or more other nodes of the computing nodes ([0099] In step S208, the optimizer 200 assumes that the target volume which is on the migration source node which is selected in step S206 and is designated in step S207 is to be migrated to a node which is the lowest in load in all the nodes, that is, the node which is the smallest in evaluation value;).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the claims of U.S. Patent No. 12050938 with the teachings of Sakashita to find the best load to migrate (see Sakashita [0105] Then, the optimizer 200 decides whether the goodness value which is calculated in step S213 is increased (step S214). In a case where the goodness value is increased (YES in step S214), the optimizer 200 sets the provisional volume migration to the variable “BestMove”).
Similar claim mappings of claims 33-40 would have been obvious to a person having ordinary skill in the art but have been omitted for the sake of brevity.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 32, 33, and 36 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Sakashita et al. (US 20210105322 A1 hereinafter Sakashita).
Sakashita was cited in the IDS filed on 07/29/2024.
As per claim 32, Sakashita teaches a computing device for correcting load imbalances in a cluster including computing nodes, the computing device comprising ([0001] The present invention relates to a distributed storage system and a data migration method and is preferably applied to the distributed storage system and the data migration method which are configured to migrate data between/among nodes in order to smooth a load; [0099] In step S208, the optimizer 200 assumes that the target volume which is on the migration source node which is selected in step S206 and is designated in step S207 is to be migrated to a node which is the lowest in load in all the nodes, that is, the node which is the smallest in evaluation value): a memory containing machine readable medium comprising machine executable code having stored thereon instructions for performing a method of load balancing in a storage system ([0142] Pieces of information such as programs, tables, files and so forth which are used to realize the respective functions may be stored into (recorded onto) storage devices such as a memory, a hard disc, an SSD (Solid State Drive) and so forth and/or recoding (storage) media; [0001] The present invention relates to a distributed storage system and a data migration method and is preferably applied to the distributed storage system and the data migration method which are configured to migrate data between/among nodes in order to smooth a load;); and a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to ([0142] all or some of the respective configurations, functions and so forth may be realized by software by interpreting and executing a program which realizes the respective functions by the processor. Pieces of information such as programs, tables, files and so forth which are used to realize the respective functions may be stored into (recorded onto) storage devices): identify a node of the computing nodes having a load that is highest of loads in the cluster; calculate work portions of volumes on the node, wherein each of the work portions represents a contribution of a corresponding volume to the load (Figs. 8 and 17; [0092] the optimizer 200 rearranges the nodes and the volumes in descending order of the evaluation values in step S202 and thereby the node and the volume which are the highest in evaluation value, in other words, the node and the volume which are the highest in current load are rearranged to high-order positions; [0097] In step S206, the optimizer 200 selects the “nodeIndex”-th node which is obtained after rearrangement of the nodes and volumes in step S202 as the migration source node; [0090] Next, the optimizer 200 executes an evaluation value calculation for calculating evaluation values of the node and the volume and rearranges the nodes and the volumes in descending order of evaluation values on the basis of a calculation result of the evaluation value calculation (step S202). Then, the optimizer 200 executes processes in step S203 and succeeding steps with arrangement of the nodes and the volumes which are rearranged in step S202 being set as a standard; [0062] The migration operator 400 observes a per-volume time-series load; [0068] the node configuration table 121 has data items of the storage node ID 1211, a processor frequency 1212, the number of processors 1213, a memory 1214, a bandwidth of internode network 1215, a bandwidth of compute network 1216, the number of drives 1217, a total capacity 1218 and an used capacity 1219 and the specification of the hardware which is loaded on each node 10 is described in the table 121. In these items, a total value of the capacities of the drives which are loaded on the target node 10 is described in the total capacity 1218 and this capacity corresponds to the capacity of the drive pool 22. In addition, the total capacity which is allocated to each volume 24 from the physical drive 21 is described in the used capacity 1219; [0069] FIG. 8 is a diagram illustrating one configuration example of the volume placement table. The volume placement table 122 is a data table which indicates such volume placement that each volume 24 is to be placed on which node 10. In the example in FIG. 8, the volume arrangement table 122 has data items of a volume ID 1221, the used capacity 1222 and the storage node ID 1223. The storage node ID 1223 in FIG. 8 corresponds to the storage node ID 1211 on the node configuration table 121 in FIG. 7.); determine a subset of the volumes to move from the node based on a performance impact of moving the work portions from the node ([0100] Then, the optimizer 200 decides whether the capacity of the volume is smaller than a predetermined threshold value (step S209). This threshold value is defined by the user or the distributed storage system 1, for example, from the viewpoints as follows. In general, it is thought that a volume which is large in capacity becomes high also in load in proportion and therefore even in a case where the volume is migrated to another node, the possibility that the volume would bring the migration destination node into an overloaded state is high. In addition, since also the cost involved in migration is high, it is difficult to obtain advantages of migration. Accordingly, the optimizer 200 defines the predetermined threshold value. In step S209, in a case where the capacity of the volume is smaller than the predetermined threshold value (YES in step S209), the optimizer 200 proceeds to step S210. In a case where the capacity of the volume is larger than the predetermined threshold value (NO in step S209), the optimizer 200 skips processes in steps up to step S215 and decides to exit from a loop in step S207. [0101] In step S210, the optimizer 200 confirms whether the migration destination node has the free capacity which is sufficient for migration of the target volume. In a case where the sufficient free capacity is present (YES in step S210), the optimizer 200 proceeds to step S211.); and move the subset of the volumes from the node to one or more other nodes of the computing nodes ([0099] In step S208, the optimizer 200 assumes that the target volume which is on the migration source node which is selected in step S206 and is designated in step S207 is to be migrated to a node which is the lowest in load in all the nodes, that is, the node which is the smallest in evaluation value;).
As per claim 33, Sakashita teaches the computing device of claim 32, wherein to determine the subset of the volumes, the processor is configured to: identify one or more of the volumes associated with those of the work portions below a threshold portion of the load; and select the subset from the one or more of the volumes ([0100] In step S209, in a case where the capacity of the volume is smaller than the predetermined threshold value (YES in step S209), the optimizer 200 proceeds to step S210. [0101] In step S210, the optimizer 200 confirms whether the migration destination node has the free capacity which is sufficient for migration of the target volume. In a case where the sufficient free capacity is present (YES in step S210), the optimizer 200 proceeds to step S211).
As per claim 36, Sakashita teaches a non-transitory machine readable medium having stored thereon instructions for correcting load imbalances in a cluster including computing nodes, the instructions comprising machine executable code that, when executed by at least one machine, causes the at least one machine to ([0142] Incidentally, all or some of the respective configurations, functions, processing units, processing sections and so forth in the above-described embodiment may be realized by hardware, for example, by designing all or some of them by an integrated circuit and/or other methods. In addition, all or some of the respective configurations, functions and so forth may be realized by software by interpreting and executing a program which realizes the respective functions by the processor. Pieces of information such as programs, tables, files and so forth which are used to realize the respective functions may be stored into (recorded onto) storage devices such as a memory, a hard disc, an SSD (Solid State Drive) and so forth and/or recoding (storage) media such as an IC (Integrated Circuit) card, an SD (Secure Digital) card, a DVD (Digital Versatile Disc) and so forth; [0001] The present invention relates to a distributed storage system and a data migration method and is preferably applied to the distributed storage system and the data migration method which are configured to migrate data between/among nodes in order to smooth a load; [0099] In step S208, the optimizer 200 assumes that the target volume which is on the migration source node which is selected in step S206 and is designated in step S207 is to be migrated to a node which is the lowest in load in all the nodes, that is, the node which is the smallest in evaluation value.): identify nodes of the computing nodes having loads at least a threshold amount higher than other nodes of the computing nodes; calculate work portions of volumes on the nodes, wherein the work portions comprise headroom values representing used performance capacity attributable to the volumes (Figs. 8 and 17; [0092] the optimizer 200 rearranges the nodes and the volumes in descending order of the evaluation values in step S202 and thereby the node and the volume which are the highest in evaluation value, in other words, the node and the volume which are the highest in current load are rearranged to high-order positions; [0097] In step S206, the optimizer 200 selects the “nodeIndex”-th node which is obtained after rearrangement of the nodes and volumes in step S202 as the migration source node; [0090] Next, the optimizer 200 executes an evaluation value calculation for calculating evaluation values of the node and the volume and rearranges the nodes and the volumes in descending order of evaluation values on the basis of a calculation result of the evaluation value calculation (step S202). Then, the optimizer 200 executes processes in step S203 and succeeding steps with arrangement of the nodes and the volumes which are rearranged in step S202 being set as a standard; [0069] FIG. 8 is a diagram illustrating one configuration example of the volume placement table. The volume placement table 122 is a data table which indicates such volume placement that each volume 24 is to be placed on which node 10. In the example in FIG. 8, the volume arrangement table 122 has data items of a volume ID 1221, the used capacity 1222 and the storage node ID 1223; [0062] The migration operator 400 observes a per-volume time-series load;); determine a subset of the volumes to move from the nodes based on the work portions and a performance impact of moving the volumes from the nodes ([0100] Then, the optimizer 200 decides whether the capacity of the volume is smaller than a predetermined threshold value (step S209). This threshold value is defined by the user or the distributed storage system 1, for example, from the viewpoints as follows. In general, it is thought that a volume which is large in capacity becomes high also in load in proportion and therefore even in a case where the volume is migrated to another node, the possibility that the volume would bring the migration destination node into an overloaded state is high. In addition, since also the cost involved in migration is high, it is difficult to obtain advantages of migration. Accordingly, the optimizer 200 defines the predetermined threshold value. In step S209, in a case where the capacity of the volume is smaller than the predetermined threshold value (YES in step S209), the optimizer 200 proceeds to step S210. In a case where the capacity of the volume is larger than the predetermined threshold value (NO in step S209), the optimizer 200 skips processes in steps up to step S215 and decides to exit from a loop in step S207. [0101] In step S210, the optimizer 200 confirms whether the migration destination node has the free capacity which is sufficient for migration of the target volume. In a case where the sufficient free capacity is present (YES in step S210), the optimizer 200 proceeds to step S211.); and move the subset of the volumes from the nodes to one or more of the other nodes ([0099] In step S208, the optimizer 200 assumes that the target volume which is on the migration source node which is selected in step S206 and is designated in step S207 is to be migrated to a node which is the lowest in load in all the nodes, that is, the node which is the smallest in evaluation value;).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 21, 22, 24-26 are rejected under 35 U.S.C. 103 as being unpatentable over Sakashita et al. (US 20210105322 A1 hereinafter Sakashita), in view of Oe et al. (US 20100235604 A1 hereinafter Oe), and further in view of Kachare et al. (US 20190272012 A1 hereinafter Kachare).
As per claim 21, Sakashita teaches a method for correcting load imbalances in a cluster including computing nodes, the method comprising ([0001] The present invention relates to a distributed storage system and a data migration method and is preferably applied to the distributed storage system and the data migration method which are configured to migrate data between/among nodes in order to smooth a load; [0099] In step S208, the optimizer 200 assumes that the target volume which is on the migration source node which is selected in step S206 and is designated in step S207 is to be migrated to a node which is the lowest in load in all the nodes, that is, the node which is the smallest in evaluation value.):
identifying a node of the computing nodes having a load that is highest of loads in the cluster; calculating work portions of volumes on the node (Figs. 8 and 17; [0092] the optimizer 200 rearranges the nodes and the volumes in descending order of the evaluation values in step S202 and thereby the node and the volume which are the highest in evaluation value, in other words, the node and the volume which are the highest in current load are rearranged to high-order positions; [0097] In step S206, the optimizer 200 selects the “nodeIndex”-th node which is obtained after rearrangement of the nodes and volumes in step S202 as the migration source node; [0090] Next, the optimizer 200 executes an evaluation value calculation for calculating evaluation values of the node and the volume and rearranges the nodes and the volumes in descending order of evaluation values on the basis of a calculation result of the evaluation value calculation (step S202). Then, the optimizer 200 executes processes in step S203 and succeeding steps with arrangement of the nodes and the volumes which are rearranged in step S202 being set as a standard; [0069] FIG. 8 is a diagram illustrating one configuration example of the volume placement table. The volume placement table 122 is a data table which indicates such volume placement that each volume 24 is to be placed on which node 10. In the example in FIG. 8, the volume arrangement table 122 has data items of a volume ID 1221, the used capacity 1222 and the storage node ID 1223.);
identifying a candidate subset of the volumes, wherein the candidate subset includes one or more of the volumes associated with a subset of the work portions (Fig. 17; [0097] In step S206, the optimizer 200 selects the “nodeIndex”-th node which is obtained after rearrangement of the nodes and volumes in step S202 as the migration source node; [0100] In step S209, in a case where the capacity of the volume is smaller than the predetermined threshold value (YES in step S209), the optimizer 200 proceeds to step S210. [0101] In step S210, the optimizer 200 confirms whether the migration destination node has the free capacity which is sufficient for migration of the target volume. In a case where the sufficient free capacity is present (YES in step S210), the optimizer 200 proceeds to step S211; [0069] FIG. 8 is a diagram illustrating one configuration example of the volume placement table. The volume placement table 122 is a data table which indicates such volume placement that each volume 24 is to be placed on which node 10. In the example in FIG. 8, the volume arrangement table 122 has data items of a volume ID 1221, the used capacity 1222 and the storage node ID 1223.);
determining a move subset from the candidate subset based on a performance impact of moving respective volumes in the candidate subset from the node ([0105] Then, the optimizer 200 decides whether the goodness value which is calculated in step S213 is increased (step S214). In a case where the goodness value is increased (YES in step S214), the optimizer 200 sets the provisional volume migration to the variable “BestMove”; [0104] Further, the optimizer 200 calculates the goodness value (step S213). The goodness value is obtained by subtracting the evaluation value which is obtained after migration from the evaluation value of the entire distributed storage system 1 which is obtained before migration; [0100] Then, the optimizer 200 decides whether the capacity of the volume is smaller than a predetermined threshold value (step S209). This threshold value is defined by the user or the distributed storage system 1, for example, from the viewpoints as follows. In general, it is thought that a volume which is large in capacity becomes high also in load in proportion and therefore even in a case where the volume is migrated to another node, the possibility that the volume would bring the migration destination node into an overloaded state is high. In addition, since also the cost involved in migration is high, it is difficult to obtain advantages of migration. Accordingly, the optimizer 200 defines the predetermined threshold value. In step S209, in a case where the capacity of the volume is smaller than the predetermined threshold value (YES in step S209), the optimizer 200 proceeds to step S210. In a case where the capacity of the volume is larger than the predetermined threshold value (NO in step S209), the optimizer 200 skips processes in steps up to step S215 and decides to exit from a loop in step S207. [0101] In step S210, the optimizer 200 confirms whether the migration destination node has the free capacity which is sufficient for migration of the target volume. In a case where the sufficient free capacity is present (YES in step S210), the optimizer 200 proceeds to step S211.); and
moving the move subset from the node to one or more other nodes of the computing nodes ([0099] In step S208, the optimizer 200 assumes that the target volume which is on the migration source node which is selected in step S206 and is designated in step S207 is to be migrated to a node which is the lowest in load in all the nodes, that is, the node which is the smallest in evaluation value; [0107] In step S216, in a case where “BestMove” is present (YES in step S216), the optimizer 200 adds “BestMove” concerned to “MoveCandidate” (step S217)).
Sakashita fails to teach wherein each of the work portions represents a percentage of the load; a subset of the work portions representing percentages of the load below a threshold percentage of the load.
However, Oe teaches wherein each of the work portions represents a percentage of the load ([0120] In addition, information containing "441" times/second as the number of reads (at peak), "22" times/second as the number of L0 reads (during measurement), (22/441).times.100="5%" as an L0 load (read), "45" times/second as the number of L1 reads (during measurement), (45/441).times.100="10%" as an L1 load (read), "18" times/second as the number of L2 reads (during measurement), and (18/441).times.100="4%" as an L2 load (read) is set. [0121] Furthermore, information containing "320" times/second as the number of writes (at peak), "7" times/second as the number of L0 writes (during measurement), (7/320).times.100="2%" as an L0 load (write), "20" times/second as the number of L1 writes (during measurement), (20/320).times.100="6%" as an L1 load (write), "6" times/second as the number of L2 writes (during measurement), and (6/320).times.100="2%" as an L2 load (write) is set. [0122] As a result, the L0 load (read) of "5%" and the L0 load (write) of "2%" is added up and "7%" is set to the L0 load (read+write). [0123] In addition, the L1 load (read) of "10%" and the L1 load (write) of "6%" is added up and "16%" is set to the L1 load (read+write). [0124] Furthermore, the L2 load (read) of "4%" and the L2 load (write) of "2%" is added up and "6%" is set to the L2 load (read+write). [0125] In this manner, IO size-specific read+write loads are calculated for the logical volumes 910, 920, and 930, and the read+write loads are further added up. Accordingly, breakdowns of the loads on the logical volumes 910, 920, and 930 may be evaluated; [0040] a load generated at the logical volumes A, B, and C of the storage node 2; [0065] The logical volume 910 is assigned a logical volume ID of "L0". [0066] The logical volume 920 is assigned a logical volume ID of "L1". [0067] The logical volume 930 is assigned a logical volume ID of "L2").
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined Sakashita with the teachings of Oe to prevent a shortage of resources (see Oe [0009] By monitoring storage area utilization, the occurrence of a shortage in a storage area may be prevented).
Sakashita and Oe fail to teach a subset of the work portions representing percentages of the load below a threshold percentage of the load.
However, Kachare teaches a subset of the work portions representing percentages of the load below a threshold percentage of the load ([0100] the local service provider 50 determines (S102) whether the total power consumption of the storage bank 302 is less than a first percentage threshold (e.g., 40% or a value between 30% to 50%) of the load; [0069] if a storage device 10 changes from operating at normal 61 to operating at greater than 90%; [0019] determining whether one or more second storage devices of the plurality of storage devices are consuming power under a threshold power level; [0107] According to some embodiments, the local service provider 50 identifies (S122) which storage devices 10 consume power at a level less than a threshold power level. In some examples, the threshold may be set at 75%).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined Sakashita and Oe with the teachings of Kachare to promote efficiency (see Kachare [0108] The local service provider 50 then compares the actual power consumption with threshold power level to determine if consumed power of the storage device 10 is below the threshold power level. The local service provider 50 then dynamically instructs the identified storage devices 10 to operate at a power cap corresponding to the first level (e.g., at 75% or 80% of maximum power), as opposed to the default power cap of 100% maximum power. Because the power efficiency of a PSU drops as it reaches its maximum load capacity, lowering the power cap of the storage devices 10 may bring down the overall power usage of the storage bank 302, thus allowing the PSU to operate at a lower power level and at a higher (e.g., peak) power efficiency range.).
As per claim 22, Sakashita, Oe, and Kachare teach the method of claim 21. Sakashita teaches comprising: collecting performance samples from the computing nodes over a collection time; and determining the loads over the collection time from the performance samples ([0060] In the present embodiment, the monitor 100 monitors (monitoring processing) loads on hardware such as the CPU 11, the drive 16 and so forth which are included in each node 10; [0073] FIG. 10 is a diagram illustrating one configuration example of the volume performance table 124. The volume performance table 124 is a data table which indicates performance information per volume 24 which is monitored by the monitor 100 in time series. In the example in FIG. 10, the volume performance table 124 has data items of a time stamp 1241, the volume ID 1242, a random ratio 1243, an average size 1244, a read IOPS (Input/Output operations Per Second) 1245, a write IOPS 1246, a read transfer rate 1247 and a write transfer rate 1248. the volume ID 1242 in FIG. 10 corresponds to the volume ID 1221 in FIG. 8 and the volume ID 1231 in FIG. 9; [0083] Incidentally, in regard to an execution timing of the monitoring processing, it is possible to optionally set the time period; [0044] As illustrated in FIG. 1, the distributed storage system 1 is configured by mutually connecting a plurality of nodes 10A to 10D).
As per claim 24, Sakashita, Oe, and Kachare teach the method of claim 21. Kachare teaches wherein identifying the node comprises: determining a subset of the computing nodes are operating at an optimal performance capacity; and identifying the node from the subset prior to the node exceeding the optimal performance capacity ([0100] According to some embodiments, the local service provider 50 manages (e.g., optimizes) operations of the PDU 90 by dynamically monitoring the operation of the PSUs 304 of the PDU 90 and ensuring that active PSUs 304 operate in their high power-efficiency range; [0108] The local service provider 50 then compares the actual power consumption with threshold power level to determine if consumed power of the storage device 10 is below the threshold power level. The local service provider 50 then dynamically instructs the identified storage devices 10 to operate at a power cap corresponding to the first level (e.g., at 75% or 80% of maximum power), as opposed to the default power cap of 100% maximum power. Because the power efficiency of a PSU drops as it reaches its maximum load capacity, lowering the power cap of the storage devices 10 may bring down the overall power usage of the storage bank 302, thus allowing the PSU to operate at a lower power level and at a higher (e.g., peak) power efficiency range.).
As per claim 25, Sakashita, Oe, and Kachare teach the method of claim 24. Sakashita teaches an estimated performance capacity for the computing nodes ([0076] FIG. 13 is a diagram illustrating one configuration example of the predicted performance table 127. The predicted performance table 1227 is a data table which describes the time-series predicted performance per volume 24; [0113] In FIG. 20, first, the performance simulator 300 calculates a predicted value of the resource utilization rate (step S301). The resource utilization rate has a value which is obtained by dividing each IOPS (the read IOPS 1245 and the write IOPS 1246 on the volume performance table 124) at a certain time (per time) by a maximum IOPS of each node per resource of each node. Here, it is possible to obtain the maximum IOPS of each node by using a tool which calculates an expected IOPS from a hardware specification such as a sizing toll while referring to the node configuration table 121).
Additionally, Kachare teaches wherein the optimal performance capacity comprises a portion of a performance capacity for the computing nodes ([0108] The local service provider 50 then compares the actual power consumption with threshold power level to determine if consumed power of the storage device 10 is below the threshold power level. The local service provider 50 then dynamically instructs the identified storage devices 10 to operate at a power cap corresponding to the first level (e.g., at 75% or 80% of maximum power), as opposed to the default power cap of 100% maximum power. Because the power efficiency of a PSU drops as it reaches its maximum load capacity, lowering the power cap of the storage devices 10 may bring down the overall power usage of the storage bank 302, thus allowing the PSU to operate at a lower power level and at a higher (e.g., peak) power efficiency range.).
As per claim 26, Sakashita, Oe, and Kachare teach the method of claim 21. Sakashita teaches wherein identifying the node comprises: determining a subset of the computing nodes are operating above an optimal performance capacity ([0092] the optimizer 200 rearranges the nodes and the volumes in descending order of the evaluation values in step S202 and thereby the node and the volume which are the highest in evaluation value, in other words, the node and the volume which are the highest in current load are rearranged to high-order positions; [0097] In step S206, the optimizer 200 selects the “nodeIndex”-th node which is obtained after rearrangement of the nodes and volumes in step S202 as the migration source node; [0088] FIG. 17 is a flowchart (a part 1) illustrating one processing procedure example of optimum placement decision processing).
Additionally, Kachare teaches identifying the node to bring the load back to the optimal performance capacity ([0102] At that point, the local service provider 50 proceeds to determine (S108) whether the total power consumption of the storage bank 302 is greater than a second percentage threshold (e.g., about 90% or a value between 85% and 95%) of the load of each of the active PSUs 304. If so, the active PSUs 304 may be operating in high-power state, which may be detrimental to the longevity of the PSUs 304 if prolonged. As such, the local service provider 50 enables (i.e., activates) a disabled (i.e., a deactivated) PSU 304 (S110), waits (S112) for a period of time (e.g., seconds or minutes), and rechecks (S108) whether the total power consumption of the storage bank 302 is still equal to or greater than the second percentage threshold of the load of each of the active PSUs 304. If so, the loop continues and the local service provider 50 continues to enable the active PSUs 304 one by one until the total power consumption is less than the second percentage threshold of the load of each of the active PSUs 304.).
Claims 23 and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Sakashita, Oe, and Kachare, as applied to claim 21 above, in view of Sampathkumar et al. (US 20180097874 A1 hereinafter Sampathkumar).
As per claim 23, Sakashita, Oe, and Kachare teach the method of claim 21.
Sakashita, Oe, and Kachare fail to teach wherein identifying the node comprises: detecting a threshold difference between the load and at least one other of the loads in the cluster; and selecting the node in response to detection of the threshold difference.
However, Sampathkumar teaches wherein identifying the node comprises: detecting a threshold difference between the load and at least one other of the loads in the cluster; and selecting the node in response to detection of the threshold difference ([0014] On the other hand, if the difference in resource utilization between the most loaded host and the least loaded host exceeds the threshold difference, workloads at the most loaded host are evaluated for migration to the least loaded host i).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined Sakashita, Oe, and Kachare with the teachings of Sampathkumar to provide better load balancing (see Sampathkumar [0021] FIG. 2 illustrates a flow diagram of method 200 for load balancing hosts in a cluster based on pairwise determined differences in resource utilization; [0002] Resource schedulers generally use standard deviation of resource utilization among multiple hosts in the cluster as a trigger for performing load balancing. In some cases, however, the standard deviation approach may not be sufficient to identify all of the load balancing opportunities. For example, where there are a small number of outliers in the cluster (e.g., hosts having a very high resource utilization relative to the average), load balancing opportunities for such outliers may be missed because the standard deviation may still be below the threshold required to trigger load balancing.).
As per claim 27, Sakashita, Oe, and Kachare teach the method of claim 21.
Sakashita, Oe, and Kachare fail to teach comprising: after moving the move subset, identifying a subsequent node of the computing nodes having a next load that is highest of loads in the cluster; and identifying and moving a second move subset from the subsequent node.
However, Sampathkumar teaches comprising: after moving the move subset, identifying a subsequent node of the computing nodes having a next load that is highest of loads in the cluster; and identifying and moving a second move subset from the subsequent node ([0022] At step 220, resource scheduler 110 selects a first host and a second host for examination. In the illustrated embodiment, resource scheduler 110 selects the most loaded host (e.g., the first host) and the least loaded host (e.g., the second host); [0023] On the other hand, if resource scheduler 110 determines that the resource utilization difference between the first and second hosts exceeds the threshold difference, resource scheduler 110 triggers load balancing between the most loaded host and the least loaded host. At step 240, resource scheduler 110 selects one or more candidate workloads for migration from the most loaded host to the least loaded host; [0024] After step 250, method 200 returns to step 220, where resource scheduler 110 selects the next two hosts for examination. In the embodiment, the next most loaded host and the next least loaded host are selected. The steps after 220 are carried out in the same manner as described above. So long as the resource utilization difference between the two hosts selected in step 220 exceed the threshold difference (as determined at step 230), method 200 continues to recommend workload migrations).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined Sakashita, Oe, and Kachare with the teachings of Sampathkumar to provide better load balancing (see Sampathkumar [0021] FIG. 2 illustrates a flow diagram of method 200 for load balancing hosts in a cluster based on pairwise determined differences in resource utilization; [0002] Resource schedulers generally use standard deviation of resource utilization among multiple hosts in the cluster as a trigger for performing load balancing. In some cases, however, the standard deviation approach may not be sufficient to identify all of the load balancing opportunities. For example, where there are a small number of outliers in the cluster (e.g., hosts having a very high resource utilization relative to the average), load balancing opportunities for such outliers may be missed because the standard deviation may still be below the threshold required to trigger load balancing.).
Claims 28 and 29 are rejected under 35 U.S.C. 103 as being unpatentable over Sakashita, Oe, and Kachare, as applied to claim 21 above, in view of Cadambi et al. (US 20140237477 A1 hereinafter Cadambi).
As per claim 28, Sakashita, Oe, and Kachare teach the method of claim 21. Sakashita teaches comprising: identifying compatible nodes of the computing nodes to accept the move subset ([0101] In step S210, the optimizer 200 confirms whether the migration destination node has the free capacity which is sufficient for migration of the target volume. In a case where the sufficient free capacity is present (YES in step S210), the optimizer 200 proceeds to step S211.).
Sakashita, Oe, and Kachare fail to teach after eliminating a portion of the compatible nodes that do not meet performance requirements of the move subset, selecting the one or more other nodes from the compatible nodes.
However, Cadambi teaches after eliminating a portion of the compatible nodes that do not meet performance requirements of the move subset, selecting the one or more other nodes from the compatible nodes ([0139] When a task with a deadline and resource requirements arrives at the cluster, the cluster scheduler send the task requirements to each node and queries if the node can accept the task. Nodes reject tasks if they do not have sufficient resources, but otherwise indicate they can accept the task with an estimated completion time and confidence level. The cluster scheduler then issues the task to a suitable node, or rejects the task if resources are insufficient or if it cannot complete the task within its deadline due to system load; [0140] The node-level scheduler schedules tasks and their offloads using a novel aging and criticality-based heuristic. Aging guarantees fairness, while criticality, which depends on deadlines and processing times, attempts to prioritize tasks and offloads so that maximal deadlines are met; abstract creating a list of nodes that have sufficient free resources at a present time to satisfy the job requirements; and assigning the job to a node).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined Sakashita, Oe, and Kachare with the teachings of Cadambi to provide nodes with sufficient resources that can meet performance requirements (see Cadambi abstract creating a list of nodes that have sufficient free resources at a present time to satisfy the job requirements; and assigning the job to a node).
As per claim 29, Sakashita, Oe, Kachare, and Cadambi teach the method of claim 28. Sakashita teaches comprising: determining an estimated performance impact to the compatible nodes based on projected used performance capacities of the compatible nodes should the move subset be moved thereto ([0101] In step S210, the optimizer 200 confirms whether the migration destination node has the free capacity which is sufficient for migration of the target volume. In a case where the sufficient free capacity is present (YES in step S210), the optimizer 200 proceeds to step S211; [0102] In step S211, the optimizer 200 calls the performance simulator 300…The performance simulator 300 which is called executes performance simulation processing on the basis of the respective pieces of information which are input and outputs predicted information on the performance (the predicted performance table 127) and predicted information on the resource utilization rate (the resource utilization table 126) to the optimizer 200 as a result of execution of the performance simulation processing; claim 1 estimates a per-period usage of each resource on a migration destination node when the data is migrated to the migration destination node by using the acquired per-data resource usage and calculates an estimate of a latency from the estimated usage of each resource, and decides a migration pattern of the data on the basis of the estimate of the latency.).
Additionally, Cadambi teaches removing another portion of the compatible nodes based on the estimated performance impact prior to selecting the one or more other nodes ([0044] The node schedulers 42 respond by indicating they can accept or reject the task. If they indicate they can accept the task, an estimated completion time along with a confidence level is provided. The cluster scheduler 40 uses this information to select a node 44 to which the task can be dispatched, or rejects the task; [0102] From among all nodes that accept the task, the cluster scheduler 40 obtains the subset of nodes L1 whose confidence level is above a cluster administrator-specified threshold at block 578. From L1, it obtains the set of nodes L2 whose estimated completion times are earlier than the task deadline at block 580. From among the nodes in L2, the task is assigned and dispatched to the node with the earliest estimated completion time at block 584. If no node meets the above criteria at block 582, the cluster scheduler selects the node m' whose confidence level is above the threshold and whose estimated completion time est.sub.ij.sup.m' is the latest).
Claim 30 is rejected under 35 U.S.C. 103 as being unpatentable over Sakashita, Oe, and Kachare, as applied to claim 21 above, in view of Prahlad et al. (US 20060053263 A1 hereinafter Prahlad).
As per claim 30, Sakashita, Oe, and Kachare teach the method of claim 21.
Sakashita, Oe, and Kachare fails to teach comprising: after moving the move subset, selecting a portion of volumes remaining on the node as candidates for imposing limits; and implementing a Quality of Service (QoS) policy on the portion of the volumes, wherein the QoS policy limits growth of the load.
However, Prahlad teaches comprising: after moving the move subset, selecting a portion of volumes remaining on the node as candidates for imposing limits; and implementing a Quality of Service (QoS) policy on the portion of the volumes, wherein the QoS policy limits growth of the load ([0136] SRM data related to the total amount of disk space remaining in primary volume 190 thus may be evaluated against a threshold or other criteria. For example, a service level agreement ("SLA") may require that primary volume 190 satisfy a threshold of having 20% available free space to guard against failure due to lack of storage capacity. HSM data related to secondary storage may also be evaluated against a threshold or other criteria. For example, an SLA or an administrator preference may require that data older than a given time period be migrated from secondary storage volume 211 to other storage or from primary volume 191 to secondary storage volume 211).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined Sakashita, Oe, and Kachare with the teachings of Prahlad to avoid failure (see Prahlad [0136] a service level agreement ("SLA") may require that primary volume 190 satisfy a threshold of having 20% available free space to guard against failure due to lack of storage capacity.).
Claim 31 is rejected under 35 U.S.C. 103 as being unpatentable over Sakashita, Oe, Kachare, and Prahlad, as applied to claim 30 above, in view of Matsuda et al. (US 20210286641 A1 hereinafter Matsuda).
As per claim 31, Sakashita, Oe, Kachare, and Prahlad teach the method of claim 30.
Sakashita, Oe, Kachare, and Prahlad fail to teach comprising: after a predefined time period, evaluating effect of the QoS policy on the portion of the volumes; and when the effect has not affected the load as expected, modifying the QoS policy.
However, Matsuda teaches comprising: after a predefined time period, evaluating effect of the QoS policy on the portion of the volumes; and when the effect has not affected the load as expected, modifying the QoS policy ([0077] The adjustment unit 14 adjusts the QoS value of the migration destination physical volume 6b on the basis of load conditions of the migration source storage device 6; [0129] Furthermore, the information processing system 1 resets the QoS value of the route and the QoS value of the migration destination physical volume 6b at the second time interval; [0137] Then, the control unit 13 generates a command to set the data transfer route on the basis of the QoS value determined by the adjustment unit 14 and issues the command to the main FC switch 3. Therefore, the information processing system 1 can suppress an increase in the load of the port used for data transfer.).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined Sakashita, Oe, Kachare, and Prahlad with the teachings of Matsuda to reduce the load on a resource (see Matsuda [0137] Then, the control unit 13 generates a command to set the data transfer route on the basis of the QoS value determined by the adjustment unit 14 and issues the command to the main FC switch 3. Therefore, the information processing system 1 can suppress an increase in the load of the port used for data transfer).
Claims 34 and 37 are rejected under 35 U.S.C. 103 as being unpatentable over Sakashita, as applied to claims 32 and 36 above, in view of Sampathkumar.
As per claim 34, Sakashita teaches the computing device of claim 32, wherein to identify the node, the processor is configured to: collect performance samples from the computing nodes over a collection time; determine the loads over the collection time from the performance samples ([0060] In the present embodiment, the monitor 100 monitors (monitoring processing) loads on hardware such as the CPU 11, the drive 16 and so forth which are included in each node 10; [0073] FIG. 10 is a diagram illustrating one configuration example of the volume performance table 124. The volume performance table 124 is a data table which indicates performance information per volume 24 which is monitored by the monitor 100 in time series. In the example in FIG. 10, the volume performance table 124 has data items of a time stamp 1241, the volume ID 1242, a random ratio 1243, an average size 1244, a read IOPS (Input/Output operations Per Second) 1245, a write IOPS 1246, a read transfer rate 1247 and a write transfer rate 1248. the volume ID 1242 in FIG. 10 corresponds to the volume ID 1221 in FIG. 8 and the volume ID 1231 in FIG. 9; [0083] Incidentally, in regard to an execution timing of the monitoring processing, it is possible to optionally set the time period; [0044] As illustrated in FIG. 1, the distributed storage system 1 is configured by mutually connecting a plurality of nodes 10A to 10D).
Sakashita fails to teach detect a threshold difference between the load and at least one other of the loads in the cluster; and select the node in response to detection of the threshold difference.
However, Sampathkumar teaches detect a threshold difference between the load and at least one other of the loads in the cluster; and select the node in response to detection of the threshold difference ([0014] On the other hand, if the difference in resource utilization between the most loaded host and the least loaded host exceeds the threshold difference, workloads at the most loaded host are evaluated for migration to the least loaded host i).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined Sakashita with the teachings of Sampathkumar to provide better load balancing (see Sampathkumar [0021] FIG. 2 illustrates a flow diagram of method 200 for load balancing hosts in a cluster based on pairwise determined differences in resource utilization; [0002] Resource schedulers generally use standard deviation of resource utilization among multiple hosts in the cluster as a trigger for performing load balancing. In some cases, however, the standard deviation approach may not be sufficient to identify all of the load balancing opportunities. For example, where there are a small number of outliers in the cluster (e.g., hosts having a very high resource utilization relative to the average), load balancing opportunities for such outliers may be missed because the standard deviation may still be below the threshold required to trigger load balancing.).
As per claim 37, Sakashita teaches the non-transitory machine readable medium of claim 36, wherein the loads are determined over a predefined time ([0060] In the present embodiment, the monitor 100 monitors (monitoring processing) loads on hardware such as the CPU 11, the drive 16 and so forth which are included in each node 10; [0073] FIG. 10 is a diagram illustrating one configuration example of the volume performance table 124. The volume performance table 124 is a data table which indicates performance information per volume 24 which is monitored by the monitor 100 in time series. In the example in FIG. 10, the volume performance table 124 has data items of a time stamp 1241, the volume ID 1242, a random ratio 1243, an average size 1244, a read IOPS (Input/Output operations Per Second) 1245, a write IOPS 1246, a read transfer rate 1247 and a write transfer rate 1248. the volume ID 1242 in FIG. 10 corresponds to the volume ID 1221 in FIG. 8 and the volume ID 1231 in FIG. 9; [0083] Incidentally, in regard to an execution timing of the monitoring processing, it is possible to optionally set the time period).
Sakashita fails to teach wherein the threshold amount is a difference in load.
However, Sampathkumar teaches wherein the threshold amount is a difference in load ([0014] On the other hand, if the difference in resource utilization between the most loaded host and the least loaded host exceeds the threshold difference, workloads at the most loaded host are evaluated for migration to the least loaded host i).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined Sakashita with the teachings of Sampathkumar to provide better load balancing (see Sampathkumar [0021] FIG. 2 illustrates a flow diagram of method 200 for load balancing hosts in a cluster based on pairwise determined differences in resource utilization; [0002] Resource schedulers generally use standard deviation of resource utilization among multiple hosts in the cluster as a trigger for performing load balancing. In some cases, however, the standard deviation approach may not be sufficient to identify all of the load balancing opportunities. For example, where there are a small number of outliers in the cluster (e.g., hosts having a very high resource utilization relative to the average), load balancing opportunities for such outliers may be missed because the standard deviation may still be below the threshold required to trigger load balancing.).
Claim 35 is rejected under 35 U.S.C. 103 as being unpatentable over Sakashita, as applied to claim 32 above, in view of Faulkner et al. (US 20160259742 A1 hereinafter Faulkner).
As per claim 35, Sakashita teaches the computing device of claim 32.
Sakashita fails to teach wherein the processor is configured to: identify a workload on a second node of the computing nodes that is growing at an abnormal rate; and implementing a Quality of Service (QoS) limit on the workload, wherein the QoS limit throttles the workload to bring the second node into an optimal performance range.
However, Faulkner teaches wherein the processor is configured to: identify a workload on a second node of the computing nodes that is growing at an abnormal rate ([0168] As mentioned above, this determination is made, when a workload has an abnormally high visit rate, high abnormal service time and/or high utilization with respect to an expected range; The data object type on which a workload is defined, for example, one of vserver, volume, LUN, file or node; Table III The data object type on which a workload is defined, for example, one of vserver, volume, LUN, file or node; [0072] Each of the nodes 208.1-208.3 is defined as a computing system to provide application services); and implementing a Quality of Service (QoS) limit on the workload, wherein the QoS limit throttles the workload to bring the second node into an optimal performance range ([0140] storage volume that was throttled by a QOS policy limit; [0010] monitors QOS data for the storage volume for determining whether a current QOS data for the storage volume is within the expected range; [0081] Performance manager 121 compares historical QOS data with current QOS data to identify a victim workload whose performance may have decreased. Victim workloads may be identified based on response time deviation from an expected response time, as described below. After identifying the victim, the performance manager 121 identifies the resource that may be in contention as well as the workloads (or volumes) that may be overusing the resources (i.e. bully workloads). Workloads are ranked to determine which bullies have the highest change in usage of the resource and which victims are most impacted. Based on the identification of victim and bully workloads, a remediation plan may be recommended to correct the problems associated with the incident; [0136] It is noteworthy that there is correlation between average latency and the number of IOPS. If the number of IOPS drops significantly below the first threshold value, then the storage system may re-allocate resources to other workloads that have higher or regular number of IOPS.).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined Sakashita with the teachings of Faulkner to promote efficiency (see Faulkner [0173] In one aspect, performance manager 121 provides efficient methods and systems for collecting QOS data, monitoring QOS data, dynamically predicting expected behavior of the QOS (i.e. the expected range) and using the historical data to identify incidents that a client may want to address. The performance 121 also analyzes each incident and provides useful recommendations to clients such that clients can reach their storage related goals and objectives.).
Claims 38 and 39 are rejected under 35 U.S.C. 103 as being unpatentable over Sakashita, as applied to claim 36 above, in view of Cadambi.
As per claim 38, Sakashita teaches the non-transitory machine readable medium of claim 36, wherein the machine executable code causes the at least one machine to: identify compatible nodes of the other nodes to accept the subset of the volumes ([0101] In step S210, the optimizer 200 confirms whether the migration destination node has the free capacity which is sufficient for migration of the target volume. In a case where the sufficient free capacity is present (YES in step S210), the optimizer 200 proceeds to step S211.).
Sakashita fails to teach after a portion of the compatible nodes are eliminated for not meeting performance requirements of the subset of the volumes, select the one or more of the other nodes from the compatible nodes.
However, Cadambi teaches after a portion of the compatible nodes are eliminated for not meeting performance requirements of the subset of the volumes, select the one or more of the other nodes from the compatible nodes ([0139] When a task with a deadline and resource requirements arrives at the cluster, the cluster scheduler send the task requirements to each node and queries if the node can accept the task. Nodes reject tasks if they do not have sufficient resources, but otherwise indicate they can accept the task with an estimated completion time and confidence level. The cluster scheduler then issues the task to a suitable node, or rejects the task if resources are insufficient or if it cannot complete the task within its deadline due to system load; [0140] The node-level scheduler schedules tasks and their offloads using a novel aging and criticality-based heuristic. Aging guarantees fairness, while criticality, which depends on deadlines and processing times, attempts to prioritize tasks and offloads so that maximal deadlines are met; abstract creating a list of nodes that have sufficient free resources at a present time to satisfy the job requirements; and assigning the job to a node).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined Sakashita with the teachings of Cadambi to provide nodes with sufficient resources that can meet performance requirements (see Cadambi abstract creating a list of nodes that have sufficient free resources at a present time to satisfy the job requirements; and assigning the job to a node).
As per claim 39, Sakashita and Cadambi teach the non-transitory machine readable medium of claim 38. Sakashita teaches wherein the machine executable code causes the at least one machine to: determine an estimated performance impact to the compatible nodes based on projected used performance capacities of the compatible nodes should the subset be moved thereto ([0101] In step S210, the optimizer 200 confirms whether the migration destination node has the free capacity which is sufficient for migration of the target volume. In a case where the sufficient free capacity is present (YES in step S210), the optimizer 200 proceeds to step S211; [0102] In step S211, the optimizer 200 calls the performance simulator 300…The performance simulator 300 which is called executes performance simulation processing on the basis of the respective pieces of information which are input and outputs predicted information on the performance (the predicted performance table 127) and predicted information on the resource utilization rate (the resource utilization table 126) to the optimizer 200 as a result of execution of the performance simulation processing; claim 1 estimates a per-period usage of each resource on a migration destination node when the data is migrated to the migration destination node by using the acquired per-data resource usage and calculates an estimate of a latency from the estimated usage of each resource, and decides a migration pattern of the data on the basis of the estimate of the latency.).
Additionally, Cadambi teaches remove another portion of the compatible nodes based on the estimated performance impact prior to selection of the one or more of the other nodes ([0044] The node schedulers 42 respond by indicating they can accept or reject the task. If they indicate they can accept the task, an estimated completion time along with a confidence level is provided. The cluster scheduler 40 uses this information to select a node 44 to which the task can be dispatched, or rejects the task; [0102] From among all nodes that accept the task, the cluster scheduler 40 obtains the subset of nodes L1 whose confidence level is above a cluster administrator-specified threshold at block 578. From L1, it obtains the set of nodes L2 whose estimated completion times are earlier than the task deadline at block 580. From among the nodes in L2, the task is assigned and dispatched to the node with the earliest estimated completion time at block 584. If no node meets the above criteria at block 582, the cluster scheduler selects the node m' whose confidence level is above the threshold and whose estimated completion time est.sub.ij.sup.m' is the latest).
Claim 40 is rejected under 35 U.S.C. 103 as being unpatentable over Sakashita and Cadambi, applied to claim 38 above, in view of Prahlad.
As per claim 40, Sakashita and Cadambi teach the non-transitory machine readable medium of claim 38.
Sakashita and Cadambi fail to teach wherein the machine executable code causes the at least one machine to: after the subset of the volumes is moved, select a portion of volumes remaining on the nodes as candidates for imposing limits; and implement a Quality of Service (QoS) policy on the portion of the volumes, wherein the QoS policy limits growth of the loads.
However, Prahlad teaches wherein the machine executable code causes the at least one machine to: after the subset of the volumes is moved, select a portion of volumes remaining on the nodes as candidates for imposing limits; and implement a Quality of Service (QoS) policy on the portion of the volumes, wherein the QoS policy limits growth of the loads ([0136] SRM data related to the total amount of disk space remaining in primary volume 190 thus may be evaluated against a threshold or other criteria. For example, a service level agreement ("SLA") may require that primary volume 190 satisfy a threshold of having 20% available free space to guard against failure due to lack of storage capacity. HSM data related to secondary storage may also be evaluated against a threshold or other criteria. For example, an SLA or an administrator preference may require that data older than a given time period be migrated from secondary storage volume 211 to other storage or from primary volume 191 to secondary storage volume 211).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined Sakashita and Cadambi with the teachings of Prahlad to avoid failure (see Prahlad [0136] a service level agreement ("SLA") may require that primary volume 190 satisfy a threshold of having 20% available free space to guard against failure due to lack of storage capacity.).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HSING CHUN LIN whose telephone number is (571)272-8522. The examiner can normally be reached Mon - Fri 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aimee Li can be reached at (571) 272-4169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.L./Examiner, Art Unit 2195 /Aimee Li/Supervisory Patent Examiner, Art Unit 2195
Prosecution Insights