Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Amendments
Claims 1 and 14 have been amended.
Claims 1-20 remain pending in the application.
The amendment filed 03/31/2026 is sufficient to overcome the 35 U.S.C. 103
rejections of claims 1-20 over Tariq in view of Minkin and Persson. The previous rejections have been withdrawn.
Response to Arguments
Argument 1, regarding the prior art rejections, applicant argues that none of the cited art teaches “determining, based on whether either one or both of the dimension of the input data and the dimension of the output data is a predetermined dimension, whether to convert a data arrangement scheme of either one or both of the input data and the output data of the inference operator; and converting, in response to determining to convert, the data arrangement scheme based on the determined data arrangement scheme conversion strategy”. Applicant argues that Persson is directed towards converting feature map data into sub portions of feature map data based on the selected format of subdivision irrespective of whether the dimensions of the OFM data are a predetermined dimensions, as the selection of OFM data is selected to be the predetermined dimensions.
Examiner notes this argument is moot in view of the rejections over Tariq in view of Minkin and CN 112465122 A, hereafter ‘122. ‘122 teaches determining, based on whether either one or both of the dimension of the input data and the dimension of the output data is a predetermined dimension, whether to convert a data arrangement scheme of either one or both of the input data and the output data of the inference operator (“FIG. 4 shows a schematic diagram of an exemplary neural network model to be optimized, stage 41 shows a segment of the neural network model, comprising an operator 401, operator 402, operator 403, operator 404; operator 405, wherein the operator 404 is the dimension sensitive operator, so before and after the operator 404, the operator 403 and the operator 405 is set as the dimension operator, such that the input of the neural network is NCHW format, but the input data of the operator 404 needs to be NHWC format”, page 9, paragraph 1); and converting, in response to determining to convert, the data arrangement scheme based on the determined data arrangement scheme conversion strategy (“Therefore operator 403 is a transposition operator converting NCHW into NHWC, so that the data format can be accepted by operator 404, after calculation is finished, operator 405 is converting NHWC into NCHW of the transposition operator”, page 9, paragraph 1).
The full prior art rejections are outlined below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Tariq et al (Pub. No.: US 11416959 B1), hereafter Tariq in view of Minkin (Pub. No.: US 20220309336 A1), hereafter Minkin and CN 112465122 A, hereafter ‘122.
Regarding claims 1 and 14, Tariq teaches determining whether an inference framework for a deep learning inference framework (“all of the components discussed herein can include any models, algorithms, and/or machine learning algorithms”, C8:L27-29) supports a first data arrangement scheme of a machine learning inference model (Synchronization management component 126 may determine whether or not the current model supports NCHW or NHWC formats of the processed data, C5:L23-41); determining, in response to the inference framework not supporting the first data arrangement scheme, a data arrangement scheme conversion strategy of input data and output data of an inference operator of the inference framework (Upon determining the NHWC format is not supported, synchronization management component 126 generates updated NHCW data, C5:L23-41), … and a correlation between the inference operator and the data arrangement scheme (synchronization management component 126 determines the format of data to be converted, C5:L23-41).
Tariq does not appear to explicitly teach based on a dimension of the input data received by the inference operator, a dimension of the output data output corresponding to the input data.
Minkin teaches based on a dimension of the input data received by the inference operator, a dimension of the output data output corresponding to the input data (Format may be dependent upon dimensionality of input tensor and output image, P0090).
Accordingly, it would have been obvious to a person having ordinary skill in the
art before the effective filing date of the claimed invention, having the teachings of
Tariq and Minkin before them, to include Minkin’s specific teaching of determining dimensionality of input and output data in Tariq’s method of Vision Architecture. One would have been motivated to make such a combination of determining dimensionality of input and output data for NHWC and NCHW formats (see Minkin P0090) and maintain memory locations for NHWC and NCHW formatted data to reduce a number of transpose or other operations to utilized to convert between the formats (see Tariq C5:L8-22).
Tariq in view of Minkin does not appear to explicitly teach “determining, based on whether either one or both of the dimension of the input data and the dimension of the output data is a predetermined dimension, whether to convert a data arrangement scheme of either one or both of the input data and the output data of the inference operator; and converting, in response to determining to convert, the data arrangement scheme based on the determined data arrangement scheme conversion strategy”.
‘122 teaches determining, based on whether either one or both of the dimension of the input data and the dimension of the output data is a predetermined dimension, whether to convert a data arrangement scheme of either one or both of the input data and the output data of the inference operator (“FIG. 4 shows a schematic diagram of an exemplary neural network model to be optimized, stage 41 shows a segment of the neural network model, comprising an operator 401, operator 402, operator 403, operator 404; operator 405, wherein the operator 404 is the dimension sensitive operator, so before and after the operator 404, the operator 403 and the operator 405 is set as the dimension operator, such that the input of the neural network is NCHW format, but the input data of the operator 404 needs to be NHWC format”, page 9, paragraph 1); and converting, in response to determining to convert, the data arrangement scheme based on the determined data arrangement scheme conversion strategy (“Therefore operator 403 is a transposition operator converting NCHW into NHWC, so that the data format can be accepted by operator 404, after calculation is finished, operator 405 is converting NHWC into NCHW of the transposition operator”, page 9, paragraph 1).
Accordingly, it would have been obvious to a person having ordinary skill in the
art before the effective filing date of the claimed invention, having the teachings of
Tariq, Minkin, and ‘122 before them, to include ‘122’s specific teaching of converting formats of inputs to be accepted by an operator within a neural network based on the dimension of the input in Tariq’s method of Vision Architecture. One would have been motivated to make such a combination of converting formats of inputs to be accepted by an operator within a neural network based on the dimension of the input (see ‘122 page 9, paragraph 1) and maintain memory locations for NHWC and NCHW formatted data to reduce a number of transpose or other operations to utilized to convert between the formats (see Tariq C5:L8-22) to improve the operation time of the network and reduce the consumption of the hardware resource (see ‘122 page 2, section background, paragraph 4).
Regarding claims 2 and 15, Tariq in view of Minkin and ‘122 teaches the limitations of claims 1 and 14 as outlined above. Tariq further teaches pre-processing the input data based on the dimension of the input data before inputting the input data to a first layer inference operator of the inference framework, wherein the pre-processing comprises: converting, …, the first data arrangement scheme of the input data into a second data arrangement scheme, different from the first data arrangement scheme, supported by the inference framework (Sensor data may be converted between NCHW and NHWC formats according to synchronization management component 126, C5:L8-22). Minkin further teaches in response to the dimension of the input data being the predetermined dimension (“tile size can be of a predetermined dimension”, P0090) …and the predetermined dimension being determined based on the second data arrangement scheme supported by the inference framework and the first data arrangement scheme of the machine learning inference model (“convert an image file to format suitable for inference (e.g., convert an image file to an input resolution of a machine learning model)”, P0577).
Regarding claims 3 and 16, Tariq in view of Minkin and ‘122 teaches the limitations of claims 1 and 14 as outlined above. Tariq further teaches post-processing output data output from a last layer inference operator of the inference framework, based on a dimension of the output data output from the last layer inference operator of the inference framework, wherein the post-processing comprises: converting, …, a data arrangement scheme of the data output from the last layer inference operator of the inference framework into the second data arrangement scheme supported by the machine learning inference model (When data is modified, NHWC formatted data may be converted to NCHW data according to synchronization management component 126, C5:L23-41). Minkin further teaches in response to a dimension of the data output from the last layer inference operator of the inference framework being the predetermined dimension (“tile size can be of a predetermined dimension”, P0090. “convert an image file to format suitable for inference (e.g., convert an image file to an input resolution of a machine learning model)”, P0577).
Regarding claims 4 and 17, Tariq in view of Minkin and ‘122 teaches the limitations of claims 1 and 14 as outlined above. Tariq further teaches verifying whether parameters of the inference operator are related to the data arrangement scheme of the input data and the output data, verifying whether implementation of the inference operator is not related to the data arrangement scheme of the input data and the output data (If synchronization management component 126 was related to the altering of data format, it may flag where the format was altered, C5:L23-41). Minkin further teaches verifying whether the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprise only four conditions, and the four conditions comprise: a first condition of receiving input data of the predetermined dimension and outputting output data of the predetermined dimension (Input and output data may fit a tile size of a predetermined dimension such as in the tiled technique, P0090-P0091, P0077); a second condition of receiving input data of a non-predetermined dimension and correspondingly outputting output data of the non-predetermined dimension (input and output data is not compared to any predetermined dimension such as in the image-to-column technique, P0078-P0079); a third condition of receiving the input data of the predetermined dimension and correspondingly outputting the output data of the non-predetermined dimension (Input contains one or more tiles with predetermined dimensions. Output contains im2col portions that have no predetermined dimensions, P0104, P0077); and a fourth condition of receiving the input data of the non-predetermined dimension and correspondingly outputting the output data of the predetermined dimension (Input contains im2col portions with no predetermined dimension. Output contains tiles with predetermined dimensions, P0104, P0077).
Regarding claim 5, Tariq in view of Minkin and ‘122 teaches the limitations of claim 4 as outlined above. Minkin further teaches converting the data arrangement scheme of the input data input to the inference operator into the first data arrangement scheme of the machine learning inference model in the third condition, in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the four conditions based on a result of the verifying (When the output contains portions with no predetermined dimensions and the input includes tiles that have predetermined dimensions, the output without predetermined dimensions may be combined with output that has predetermined dimensions, P0104).
Regarding claim 6, Tariq in view of Minkin and ‘122 teaches the limitations of claim 4 as outlined above. Minkin further teaches converting the data arrangement scheme of the output data of the inference operator into the second data arrangement scheme supported by the inference framework in the fourth condition, in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the four conditions based on a result of the verifying (When the output contains tiles with predetermined dimensions and the input includes portions that do not have predetermined dimensions, the output with predetermined dimensions may be combined without output that has predetermined dimensions, P0104).
Regarding claim 7, Tariq in view of Minkin and ‘122 teaches the limitations of claim 4 as outlined above. Minkin further teaches not converting the data arrangement schemes of the input data and the output data of the inference operator in the first condition and the second condition, in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the four conditions based on a result of the verifying (When there are predetermined dimensions for the input and output, dimensions of an input tensor can be used to determine how many tiles are divisible into an input tensor. In this process, the format of the data is not converted, P0092. Tiles with predetermined dimensions are used to generate the output, P0092. When there are no predetermined dimensions for input and output, such as when using the im2col technique, the portion of the input tensor does not get converted and is used to generate an output, P0093, P0096).
Regarding claims 8 and 19, Tariq in view of Minkin and ‘122 teaches the limitations of claims 1 and 14 as outlined above. Tariq further teaches verifying whether the parameters of the inference operator are related to the data arrangement scheme, verifying whether implementation of the inference operator is not related to the data arrangement scheme (If synchronization management component 126 was related to the altering of data format, it may flag where the format was altered, C5:L23-41). Minkin further teaches verifying whether the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprise only two conditions, and the two conditions comprise: a first condition of receiving input data of the predetermined dimension and outputting output data of the predetermined dimension (Input and output data may fit a tile size of a predetermined dimension, P0090-P0091); and a second condition of receiving input data of a non-predetermined dimension and correspondingly outputting output data of the non-predetermined dimension (input and output data is not compared to any predetermined dimension such as in the image-to-column technique, P0078-P0079).
Regarding claim 9, Tariq in view of Minkin and ‘122 teaches the limitations of claim 8 as outlined above. Minkin further teaches not converting the data arrangement schemes of the input data and the output data of the inference operator and adjusting the parameters of the inference operator in the second condition, in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the two conditions based on a result of the verifying (When there are predetermined dimensions for the input and output, dimensions of an input tensor can be used to determine how many tiles are divisible into an input tensor. In this process, the format of the data is not converted, P0092. Tiles with predetermined dimensions are used to generate the output, P0092).
Regarding claim 10, Tariq in view of Minkin and ‘122 teaches the limitations of claim 8 as outlined above. Minkin further teaches converting the data arrangement schemes of the input data and the output data of the inference operator and not adjusting the parameters of the inference operator in the first condition, in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the two conditions based on a result of the verifying (When there are no predetermined dimensions for input and output, such as when using the im2col technique, the portion of the input tensor does not get converted and is used to generate an output, P0093, P0096).
Regarding claim 11, Tariq in view of Minkin and ‘122 teaches the limitations of claim 1 as outlined above. Tariq further teaches determining the data arrangement scheme conversion strategy of the input data and the output data of the inference operator in response to the inference operator being executed (NHWC formatted data may be converted to NCHW data according to synchronization management component 126, C5:L23-41); or determining the data arrangement scheme conversion strategy of the input data and the output data of the inference operator prior to the inference operator being executed (Transpose and replacement of formats may be performed without synchronization management component 126 being executed, C15:L7-9).
Regarding claim 12, Tariq in view of Minkin and ‘122 teaches the limitations of claim 2 as outlined above. Minkin further teaches wherein the predetermined dimension is 4 (Tensor may be four dimensional, P0137). Tariq further teaches the first data arrangement scheme of the machine learning inference model is NHWC, and the second data arrangement scheme supported by the inference framework is NCWH, or the first data arrangement scheme of the machine learning inference model is NCWH, and the second data arrangement scheme supported by the inference framework is NHWC (NHWC formatted data may be converted to NCHW data, C5:L23-41).
Regarding claim 13, Tariq in view of Minkin and ‘122 teaches the limitations of claim 1 as outlined above. Tariq further teaches a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 (“The computer readable media 314 can store an operating system and one or more software applications, instructions, programs, and/or data to implement the methods described herein and the functions attributed to the various systems”, C10:L66-67. C11:L1-3).
Regarding claim 18, Tariq in view of Minkin and ‘122 teaches the limitations of claim 17 as outlined above. Minkin further teaches in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the four conditions based on the result of the verifying, not convert the data arrangement schemes of the input data and the output data of the inference operator in the first condition and the second condition (When there are predetermined dimensions for the input and output, dimensions of an input tensor can be used to determine how many tiles are divisible into an input tensor. In this process, the format of the data is not converted, P0092. Tiles with predetermined dimensions are used to generate the output, P0092. When there are no predetermined dimensions for input and output, such as when using the im2col technique, the portion of the input tensor does not get converted and is used to generate an output, P0093, P0096); convert the data arrangement scheme of the input data input to the inference operator into the first data arrangement scheme of the machine learning inference model in the third condition (When the output contains portions with no predetermined dimensions and the input includes tiles that have predetermined dimensions, the output without predetermined dimensions may be combined with output that has predetermined dimensions, P0104); and convert the data arrangement scheme of the output data of the inference operator into the second data arrangement scheme supported by the inference framework in the fourth condition (When the output contains tiles with predetermined dimensions and the input includes portions that do not have predetermined dimensions, the output with predetermined dimensions may be combined without output that has predetermined dimensions, P0104).
Regarding claim 20, Tariq in view of Minkin and ‘122 teaches the limitations of claim 19 as outlined above. Minkin further teaches in response to the dimension of the input data received by the inference operator and the dimension of the output data output corresponding to the input data comprising only the two conditions based on the result of the verifying, not convert the data arrangement schemes of the input data and the output data of the inference operator and not adjust the parameters of the inference operator in the first condition (When there are predetermined dimensions for the input and output, dimensions of an input tensor can be used to determine how many tiles are divisible into an input tensor. In this process, the format of the data is not converted, P0092. Tiles with predetermined dimensions are used to generate the output, P0092); and not convert the data arrangement schemes of the input data and the output data of the inference operator and adjust the parameters of the inference operator in the second condition (When there are no predetermined dimensions for input and output, such as when using the im2col technique, the portion of the input tensor does not get converted and is used to generate an output, P0093, P0096).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ISHAN MOUNDI whose telephone number is (703)756-1547. The examiner can normally be reached 8:30 A.M. - 5 P.M..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/I.M./Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights