Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments
Claims 1-2, 6-10, and 14-20 have been amended.
Claims 1-20 remain pending in the application.
The amendment filed 11/06/2025 is sufficient to overcome the 35 U.S.C. 101 rejections of claims 1-20. The previous rejections have been withdrawn.
The amendment filed 11/06/2025 is sufficient to overcome the 35 U.S.C. 103
rejections of claims 1-20 over Tariq in view of Minkin. The previous rejections have been withdrawn.
Response to Arguments
Argument 1, regarding the claim objections, applicant argues that the claim objections to claims 6-7, 9-10, 18, and 20 should be withdrawn in view of their amendments. Examiner agrees and the objections have been withdrawn.
Argument 2, regarding the 112(f) claim interpretation, applicant argues that the terms "conversion strategy determiner", "executor", "pre-processor", and "post-processor" are definite structural elements configured to perform one or more specific functions defined in the claims, but amended the claims to instead recite “processors” to advance prosecution. Examiner agrees that this language avoids 112(f) claim interpretation.
Argument 3, regarding the 101 rejections, applicant argues that the amended claims are directed towards the improvement of reducing data conversion tasks of a model in an inference operation. Examiner agrees and the 35 U.S.C. 101 rejections have been withdrawn.
Argument 4, regarding the prior art rejections, applicant argues that none of the cited art teaches “converting, in response to either one or both of the dimension of the input data and the dimension of the output data being a predetermined dimension, a data arrangement scheme of either one or both of the input data and the output data” as amended in the independent claims. Applicant argues in the case of Minkin that Minkin teaches dividing an input into one or more tiles of a predetermined dimension, but does not teach converting a data arrangement scheme in response to the input or output being a predetermined dimension. Examiner notes that this argument is moot in view of Persson et al (Pub. No.: US 20210304012 A1). Persson teaches converting, in response to either one or both of the dimension of the input data and the dimension of the output data being a predetermined dimension, a data arrangement scheme of either one or both of the input data and the output data (a format of subdivision of data is selected based on predetermined dimensions of output feature map data, P0063).
Accordingly, it would have been obvious to a person having ordinary skill in the
art before the effective filing date of the claimed invention, having the teachings of
Tariq, Minkin, and Persson before them, to include Persson’s specific teaching of selecting a format of subdivision of data based on predetermined dimensions of an output feature map in Tariq’s method of Vision Architecture. One would have been motivated to make such a combination of selecting a format of subdivision of data based on predetermined dimensions of an output feature map (see Persson P0063) and maintain memory locations for NHWC and NCHW formatted data to reduce a number of transpose or other operations to utilized to convert between the formats (see Tariq C5:L8-22).
The full prior art rejections are outlined below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Tariq et al (Pub. No.: US 11416959 B1), hereafter Tariq in view of Minkin (Pub. No.: US 20220309336 A1), hereafter Minkin and Persson et al (Pub. No.: US 20210304012 A1), hereafter Persson.
Regarding claim 14, claim limitations “one or more processors configured to…” invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. This element is interpreted under 35 U.S.C. 112(f) as processor(s) with the algorithm described in the specification (the algorithms to determine a data arrangement scheme conversion strategy and convert either a data arrangement scheme of the input data or output data of the inference operator based on the determined data arrangement scheme conversion strategy) that causes the processor(s) to perform the claimed function.
Regarding claims 1 and 14, Tariq teaches determining whether an inference framework for a deep learning inference framework (“all of the components discussed herein can include any models, algorithms, and/or machine learning algorithms”, C8:L27-29) supports a first data arrangement scheme of a machine learning inference model (Synchronization management component 126 may determine whether or not the current model supports NCHW or NHWC formats of the processed data, C5:L23-41); determining, in response to the inference framework not supporting the first data arrangement scheme, a data arrangement scheme conversion strategy of input data and output data of an inference operator of the inference framework (Upon determining the NHWC format is not supported, synchronization management component 126 generates updated NHCW data, C5:L23-41), … and a correlation between the inference operator and the data arrangement scheme (synchronization management component 126 determines the format of data to be converted, C5:L23-41).
Tariq does not appear to explicitly teach based on a dimension of the input data received by the inference operator, a dimension of the output data output corresponding to the input data.
Minkin teaches based on a dimension of the input data received by the inference operator, a dimension of the output data output corresponding to the input data (Format may be dependent upon dimensionality of input tensor and output image, P0090).
Accordingly, it would have been obvious to a person having ordinary skill in the
art before the effective filing date of the claimed invention, having the teachings of
Tariq and Minkin before them, to include Minkin’s specific teaching of determining dimensionality of input and output data in Tariq’s method of Vision Architecture. One would have been motivated to make such a combination of determining dimensionality of input and output data for NHWC and NCHW formats (see Minkin P0090) and maintain memory locations for NHWC and NCHW formatted data to reduce a number of transpose or other operations to utilized to convert between the formats (see Tariq C5:L8-22).
Tariq in view of Minkin does not appear to explicitly teach “converting, in response to either one or both of the dimension of the input data and the dimension of the output data being a predetermined dimension, a data arrangement scheme of either one or both of the input data and the output data”.
Persson teaches converting, in response to either one or both of the dimension of the input data and the dimension of the output data being a predetermined dimension, a data arrangement scheme of either one or both of the input data and the output data (a format of subdivision of data is selected based on predetermined dimensions of output feature map data, P0063).
Accordingly, it would have been obvious to a person having ordinary skill in the
art before the effective filing date of the claimed invention, having the teachings of
Tariq, Minkin, and Persson before them, to include Persson’s specific teaching of selecting a format of subdivision of data based on predetermined dimensions of an output feature map in Tariq’s method of Vision Architecture. One would have been motivated to make such a combination of selecting a format of subdivision of data based on predetermined dimensions of an output feature map (see Persson P0063) and maintain memory locations for NHWC and NCHW formatted data to reduce a number of transpose or other operations to utilized to convert between the formats (see Tariq C5:L8-22).
Regarding claim 15, claim limitation “one or more processors configured to” invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. This element is interpreted under 35 U.S.C. 112(f) as processor(s) with the algorithm described in the specification (the algorithms to pre-process input data and convert the data arrangement scheme of the input data into a second data arrangement scheme) that causes the processor(s) to perform the claimed function.
Regarding claims 2 and 15, Tariq in view of Minkin and Persson teaches the limitations of claims 1 and 14 as outlined above. Tariq further teaches pre-processing the input data based on the dimension of the input data before inputting the input data to a first layer inference operator of the inference framework, wherein the pre-processing comprises: converting, …, the first data arrangement scheme of the input data into a second data arrangement scheme, different from the first data arrangement scheme, supported by the inference framework (Sensor data may be converted between NCHW and NHWC formats according to synchronization management component 126, C5:L8-22). Minkin further teaches in response to the dimension of the input data being the predetermined dimension (“tile size can be of a predetermined dimension”, P0090) …and the predetermined dimension being determined based on the second data arrangement scheme supported by the inference framework and the first data arrangement scheme of the machine learning inference model (“convert an image file to format suitable for inference (e.g., convert an image file to an input resolution of a machine learning model)”, P0577).
Regarding claim 16, claim limitation “one or more processors configured to” invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. This element is interpreted under 35 U.S.C. 112(f) as processor(s) with the algorithm described in the specification (the algorithms to post-process output data output from a last layer inference operator of the inference framework and convert a data arrangement scheme of the data output from the last layer inference operator of the inference framework into the second data arrangement scheme) that causes the processor(s) to perform the claimed function.
Regarding claims 3 and 16, Tariq in view of Minkin and Persson teaches the limitations of claims 1 and 14 as outlined above. Tariq further teaches post-processing output data output from a last layer inference operator of the inference framework, based on a dimension of the output data output from the last layer inference operator of the inference framework, wherein the post-processing comprises: converting, …, a data arrangement scheme of the data output from the last layer inference operator of the inference framework into the second data arrangement scheme supported by the machine learning inference model (When data is modified, NHWC formatted data may be converted to NCHW data according to synchronization management component 126, C5:L23-41). Minkin further teaches in response to a dimension of the data output from the last layer inference operator of the inference framework being the predetermined dimension (“tile size can be of a predetermined dimension”, P0090. “convert an image file to format suitable for inference (e.g., convert an image file to an input resolution of a machine learning model)”, P0577).
Regarding claim 17, claim limitation “one or more processors configured to…” invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. This element is interpreted under 35 U.S.C. 112(f) as processor(s) with the algorithm described in the specification (the algorithms to verify whether parameters of the inference operator are related to the data arrangement scheme, and implementation of the inference operator is not related to the data arrangement scheme, and the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprise only four conditions, and the four conditions comprise: a first condition of receiving input data of the predetermined dimension and outputting output data of the predetermined dimension; a second condition of receiving input data of a non-predetermined dimension and correspondingly outputting output data of the non-predetermined dimension; a third condition of receiving the input data of the predetermined dimension and correspondingly outputting the output data of the non-predetermined dimension; and a fourth condition of receiving the input data of the non-predetermined dimension and correspondingly outputting the output data of the predetermined dimension) that causes the processor(s) to perform the claimed function.
Regarding claims 4 and 17, Tariq in view of Minkin and Persson teaches the limitations of claims 1 and 14 as outlined above. Tariq further teaches verifying whether parameters of the inference operator are related to the data arrangement scheme of the input data and the output data, verifying whether implementation of the inference operator is not related to the data arrangement scheme of the input data and the output data (If synchronization management component 126 was related to the altering of data format, it may flag where the format was altered, C5:L23-41). Minkin further teaches verifying whether the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprise only four conditions, and the four conditions comprise: a first condition of receiving input data of the predetermined dimension and outputting output data of the predetermined dimension (Input and output data may fit a tile size of a predetermined dimension such as in the tiled technique, P0090-P0091, P0077); a second condition of receiving input data of a non-predetermined dimension and correspondingly outputting output data of the non-predetermined dimension (input and output data is not compared to any predetermined dimension such as in the image-to-column technique, P0078-P0079); a third condition of receiving the input data of the predetermined dimension and correspondingly outputting the output data of the non-predetermined dimension (Input contains one or more tiles with predetermined dimensions. Output contains im2col portions that have no predetermined dimensions, P0104, P0077); and a fourth condition of receiving the input data of the non-predetermined dimension and correspondingly outputting the output data of the predetermined dimension (Input contains im2col portions with no predetermined dimension. Output contains tiles with predetermined dimensions, P0104, P0077).
Regarding claim 5, Tariq in view of Minkin and Persson teaches the limitations of claim 4 as outlined above. Minkin further teaches converting the data arrangement scheme of the input data input to the inference operator into the first data arrangement scheme of the machine learning inference model in the third condition, in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the four conditions based on a result of the verifying (When the output contains portions with no predetermined dimensions and the input includes tiles that have predetermined dimensions, the output without predetermined dimensions may be combined with output that has predetermined dimensions, P0104).
Regarding claim 6, Tariq in view of Minkin and Persson teaches the limitations of claim 4 as outlined above. Minkin further teaches converting the data arrangement scheme of the output data of the inference operator into the second data arrangement scheme supported by the inference framework in the fourth condition, in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the four conditions based on a result of the verifying (When the output contains tiles with predetermined dimensions and the input includes portions that do not have predetermined dimensions, the output with predetermined dimensions may be combined without output that has predetermined dimensions, P0104).
Regarding claim 7, Tariq in view of Minkin and Persson teaches the limitations of claim 4 as outlined above. Minkin further teaches not converting the data arrangement schemes of the input data and the output data of the inference operator in the first condition and the second condition, in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the four conditions based on a result of the verifying (When there are predetermined dimensions for the input and output, dimensions of an input tensor can be used to determine how many tiles are divisible into an input tensor. In this process, the format of the data is not converted, P0092. Tiles with predetermined dimensions are used to generate the output, P0092. When there are no predetermined dimensions for input and output, such as when using the im2col technique, the portion of the input tensor does not get converted and is used to generate an output, P0093, P0096).
Regarding claim 19, claim limitation “one or more processors configured to…” invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. This element is interpreted under 35 U.S.C. 112(f) as processor(s) with the algorithm described in the specification (the algorithms to verify whether the parameters of the inference operator are related to the data arrangement scheme, and implementation of the inference operator is not related to the data arrangement scheme, wherein the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprise only two conditions, and the two conditions comprise: a first condition of receiving input data of a predetermined dimension and outputting output data of the predetermined dimension; and a second condition of receiving input data of a non-predetermined dimension and correspondingly outputting output data of the non-predetermined dimension is correspondingly output) that causes the processor(s) to perform the claimed function.
Regarding claims 8 and 19, Tariq in view of Minkin and Persson teaches the limitations of claims 1 and 14 as outlined above. Tariq further teaches verifying whether the parameters of the inference operator are related to the data arrangement scheme, verifying whether implementation of the inference operator is not related to the data arrangement scheme (If synchronization management component 126 was related to the altering of data format, it may flag where the format was altered, C5:L23-41). Minkin further teaches verifying whether the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprise only two conditions, and the two conditions comprise: a first condition of receiving input data of the predetermined dimension and outputting output data of the predetermined dimension (Input and output data may fit a tile size of a predetermined dimension, P0090-P0091); and a second condition of receiving input data of a non-predetermined dimension and correspondingly outputting output data of the non-predetermined dimension (input and output data is not compared to any predetermined dimension such as in the image-to-column technique, P0078-P0079).
Regarding claim 9, Tariq in view of Minkin and Persson teaches the limitations of claim 8 as outlined above. Minkin further teaches not converting the data arrangement schemes of the input data and the output data of the inference operator and adjusting the parameters of the inference operator in the second condition, in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the two conditions based on a result of the verifying (When there are predetermined dimensions for the input and output, dimensions of an input tensor can be used to determine how many tiles are divisible into an input tensor. In this process, the format of the data is not converted, P0092. Tiles with predetermined dimensions are used to generate the output, P0092).
Regarding claim 10, Tariq in view of Minkin and Persson teaches the limitations of claim 8 as outlined above. Minkin further teaches converting the data arrangement schemes of the input data and the output data of the inference operator and not adjusting the parameters of the inference operator in the first condition, in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the two conditions based on a result of the verifying (When there are no predetermined dimensions for input and output, such as when using the im2col technique, the portion of the input tensor does not get converted and is used to generate an output, P0093, P0096).
Regarding claim 11, Tariq in view of Minkin and Persson teaches the limitations of claim 1 as outlined above. Tariq further teaches determining the data arrangement scheme conversion strategy of the input data and the output data of the inference operator in response to the inference operator being executed (NHWC formatted data may be converted to NCHW data according to synchronization management component 126, C5:L23-41); or determining the data arrangement scheme conversion strategy of the input data and the output data of the inference operator prior to the inference operator being executed (Transpose and replacement of formats may be performed without synchronization management component 126 being executed, C15:L7-9).
Regarding claim 12, Tariq in view of Minkin and Persson teaches the limitations of claim 2 as outlined above. Minkin further teaches wherein the predetermined dimension is 4 (Tensor may be four dimensional, P0137). Tariq further teaches the first data arrangement scheme of the machine learning inference model is NHWC, and the second data arrangement scheme supported by the inference framework is NCWH, or the first data arrangement scheme of the machine learning inference model is NCWH, and the second data arrangement scheme supported by the inference framework is NHWC (NHWC formatted data may be converted to NCHW data, C5:L23-41).
Regarding claim 13, Tariq in view of Minkin and Persson teaches the limitations of claim 1 as outlined above. Tariq further teaches a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 (“The computer readable media 314 can store an operating system and one or more software applications, instructions, programs, and/or data to implement the methods described herein and the functions attributed to the various systems”, C10:L66-67. C11:L1-3).
Regarding claim 18, claim limitation “one or more processors configured to…” invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. This element is interpreted under 35 U.S.C. 112(f) as processor(s) with the algorithm described in the specification (the algorithms in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the four conditions based on the result of the verifying, not convert the data arrangement schemes of the input data and the output data of the inference operator in the first condition and the second condition; convert the data arrangement scheme of the input data input to the inference operator into the first data arrangement scheme of the machine learning inference model in the third condition; and convert the data arrangement scheme of the output data of the inference operator into the second data arrangement scheme supported by the inference framework in the fourth condition) that causes the processor(s) to perform the claimed function.
Tariq in view of Minkin and Persson teaches the limitations of claim 17 as outlined above. Minkin further teaches in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the four conditions based on the result of the verifying, not convert the data arrangement schemes of the input data and the output data of the inference operator in the first condition and the second condition (When there are predetermined dimensions for the input and output, dimensions of an input tensor can be used to determine how many tiles are divisible into an input tensor. In this process, the format of the data is not converted, P0092. Tiles with predetermined dimensions are used to generate the output, P0092. When there are no predetermined dimensions for input and output, such as when using the im2col technique, the portion of the input tensor does not get converted and is used to generate an output, P0093, P0096); convert the data arrangement scheme of the input data input to the inference operator into the first data arrangement scheme of the machine learning inference model in the third condition (When the output contains portions with no predetermined dimensions and the input includes tiles that have predetermined dimensions, the output without predetermined dimensions may be combined with output that has predetermined dimensions, P0104); and convert the data arrangement scheme of the output data of the inference operator into the second data arrangement scheme supported by the inference framework in the fourth condition (When the output contains tiles with predetermined dimensions and the input includes portions that do not have predetermined dimensions, the output with predetermined dimensions may be combined without output that has predetermined dimensions, P0104).
Regarding claim 20, claim limitation “one or more processors configured to…” invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. This element is interpreted under 35 U.S.C. 112(f) as processor(s) with the algorithm described in the specification (the algorithms to response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the two conditions based on the result of the verifying, not convert the data arrangement schemes of the input data and the output data of the inference operator and not adjust the parameters of the inference operator in the first condition; and not convert the data arrangement schemes of the input data and the output data of the inference operator and adjust the parameters of the inference operator in the second condition) that causes the processor(s) to perform the claimed function.
Tariq in view of Minkin and Persson teaches the limitations of claim 19 as outlined above. Minkin further teaches in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the two conditions based on the result of the verifying, not convert the data arrangement schemes of the input data and the output data of the inference operator and not adjust the parameters of the inference operator in the first condition (When there are predetermined dimensions for the input and output, dimensions of an input tensor can be used to determine how many tiles are divisible into an input tensor. In this process, the format of the data is not converted, P0092. Tiles with predetermined dimensions are used to generate the output, P0092); and not convert the data arrangement schemes of the input data and the output data of the inference operator and adjust the parameters of the inference operator in the second condition (When there are no predetermined dimensions for input and output, such as when using the im2col technique, the portion of the input tensor does not get converted and is used to generate an output, P0093, P0096).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ISHAN MOUNDI whose telephone number is (703)756-1547. The examiner can normally be reached 8:30 A.M. - P.M..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/I.M./Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights