DETAILED ACTION
This Office Action is in response to the Preliminary Amendment entered on 01/03/2024. Claim 7 is amended. Claims 1-19 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement filed 10/18/2023 fails to comply with 37 CFR 1.98(a)(3)(i) because it does not include a concise explanation of the relevance, as it is presently understood by the individual designated in 37 CFR 1.56(c) most knowledgeable about the content of the information, of each reference listed that is not in the English language (the reference CN 111465940 does not have an English translation of the Abstract). It has been placed in the application file, but the information referred to therein has not been considered.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 8-9, 11-13, 15, 18-19 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 7-8, 12-15 of U.S. Patent 11,527,074.
Claims 2-7, 10, 14, 16-17, are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 7, 13, 21 of U.S. Patent 11,527,074 in view of Bojarski et al(NPL: “End to End Learning for Self-Driving Cars”- hereinafter Bojarski) in view of Cao et al(NPL: "DO-Conv: Depthwise Over-parameterized Convolutional Layer"- hereinafter Cao) and in further view of Alexandrovich et al(CN 116965029A - hereinafter Alexandrovich), as explained in the table below.
Although the claims at issue are not identical, they are not patentably distinct from each other because the main purpose of both the instant application and the US Patent is the same, as they are both directed towards the same inventive concept.
Instant Application
Patent No. 11,527,074
Claim 1
(Original) A computer-implemented method for automatically controlling a machine, the method comprising:
receiving data generated using at least one sensor of a machine;
performing one or more prediction tasks on the data using a neural network, wherein the neural network comprises at least one parameter tensor comprising at least one element, and the at least one parameter tensor was over-parameterized during training into a plurality of component tensors; and
controlling the machine based on results of the one or more prediction tasks.
Claim 1
A computer-implemented method for automatically controlling a vehicle, the method comprising:
receiving data generated using at least one sensor of the vehicle;
simultaneously performing multiple different prediction tasks on the data using a multi-task neural network, wherein the multi-task neural network comprises at least one shared parameter inference matrix comprising parameters shared between the multiple different prediction tasks, and the at least one shared parameter inference matrix was over-parameterized during training into at least one shared parameter matrix and multiple task-specific parameter matrices, each of the multiple task-specific parameter matrices being associated with a different one of the multiple different tasks; and
automatically controlling the vehicle based on results of the multiple different prediction tasks.
Claim 2
Claim 1
Patent cited above fails to teach the limitations at this claim, however, Alexandrovich teaches them at Paragraph 329.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teaching of Shen with the above combination of Bojarski and Cao as well as Alexandrovich to accelerate the training process.
Claim 3
Claim 1
Patent cited above fails to teach the limitations at this claim, however, Alexandrovich teaches them at Paragraph 11.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teaching of Shen with the above combination of Bojarski and Cao as well as Alexandrovich to accelerate the training process.
Claim 4
Claim 1
Patent cited above fails to teach the limitations at this claim, however, Alexandrovich teaches them at Paragraph 205.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teaching of Shen with the above combination of Bojarski and Cao as well as Alexandrovich to compensate for each error discovered during the learning process.
Claim 5
Claim 1
Patent cited above fails to teach the limitations at this claim, however, Alexandrovich teaches them at Paragraph 125.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teaching of Shen with the above combination of Bojarski and Cao as well as Alexandrovich to achieve better compression in transmission or storage.
Claim 6
Claim 2
Patent cited above fails to teach some of the limitations at this claim, however, Alexandrovich teaches them at Paragraph 200.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teaching of Shen with the teachings of Alexandrovich to mitigate the challenges posed by MLP architectures.
Claim 7
Claim 1
Patent cited above fails to teach the limitations at this claim, however, Bojarski teaches them at Page 7 Paragraph 3 Lines 1 to 3.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teaching of Shen with the above combination of Bojarski and Cao as well as Alexandrovich to achieve good performance.
Claim 8
8. (Original) A computing system for automatically controlling a machine, the computing system comprising one or more processors and memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for carrying out a computer-implemented method according to claim 1.
Claim 7
7. A computing system for automatically controlling a vehicle, the computing system comprising one or more processors and memory storing one or more programs for execution by the one or more processors for, the one or more programs including instructions for:
receiving data generated using at least one sensor of the vehicle;
simultaneously performing multiple different prediction tasks on the data using a multi-task neural network, wherein the multi-task neural network comprises at least one shared parameter inference matrix comprising parameters shared between the multiple different prediction tasks, and the at least one shared parameter inference matrix was over-parameterized during training into at least one shared parameter matrix and multiple task-specific parameter matrices, each of the multiple task-specific parameter matrices being associated with a different one of the multiple different tasks; and
automatically controlling the vehicle based on results of the multiple different prediction tasks.
Claim 9
9. (Original) The computing system of claim 8, wherein the machine is a mobile agent, and the computing system is an embedded computing system of the mobile agent.
Claim 8
8. The computing system of claim 7, wherein the computing system is an embedded computing system of the vehicle.
Claim 10
Claim 1
Patent cited above fails to teach the limitations at this claim, however, Bojarski teaches them at Page 4 Paragraph 2 Lines 1 to 2.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teaching of Shen with the above teaching of Bojarski to choose the system that has no dependencies on any particular vehicle make or model.
Claim 11
11. (Original) A computer-implemented method for generating a machine learned neural network that can perform one or more prediction tasks based on data of sensors of a machine for automatically controlling the machine, the computer-implemented method comprising:
training a learning neural network on a plurality of training datasets, the neural network comprising at least one over-parameterized parameter tensor, the at least one over-parameterized tensor comprising a plurality of component tensors; and
generating a machine learned neural network for performing one or more prediction tasks on a dataset, the machine learned neural network comprising at least one parameter tensor that is a combination of the trained plurality of component tensors;
embedding the machine learned neural network into a computing system for the machine such that the computing system performs the one or more prediction tasks on the sensor data of the machine and controls the machine based on results of the one or more prediction tasks.
Claim 13
13. A method for generating a multi-task machine learned neural network that can perform prediction tasks based on sensor data of a vehicle for automatically controlling the vehicle, the method comprising:
training a multi-task learning neural network on a plurality of training datasets, the multi-task learning neural network comprising:
a plurality of task-specific parameter sets, each task-specific parameter set dedicated to a different task, and
a shared parameter set comprising at least one over-parameterized convolutional layer, the at least one over-parameterized convolutional layer comprising at least one shared parameter matrix and a plurality of task-specific parameter matrices;
generating a multi-task machine learned neural network for simultaneously performing multiple different prediction tasks on the sensor data of the vehicle, the multi-task machine learned neural network comprising the trained plurality of task-specific parameter sets and a trained shared parameter set that comprises a matrix that is a combination of the trained at least one shared parameter matrix and the trained plurality of task-specific parameter matrices; and
embedding the multi-task machine learned neural network into a computing system for the vehicle such that the computing system can perform the multiple different prediction tasks on the sensor data of the vehicle and automatically control the vehicle based on results of the multiple different prediction tasks.
Claim 12
12. (Original) The computer-implemented method of claim 11, wherein the trained plurality of component tensors comprise an identical number of elements as the at least one parameter tensor and are compressed by element-wise addition to generate the at least one parameter tensor.
Claim 12
18. The method of claim 13, wherein the matrix multiplication of the trained at least one shared parameter matrix and the trained plurality of task-specific parameter matrices comprises an element-wise product of the trained plurality of task-specific parameter matrices.
Claim 13
13. (Original) The computer-implemented method of claim 11, wherein training the learning neural network comprises training a subset of the plurality of component tensors at each training epoch by updating elements of the subset of the plurality of components while freezing elements of any other component tensors.
Claim 15
15. The method of claim 13, wherein training the multi-task learning network comprises updating factors one of the plurality of task-specific parameter matrices using a task-specific loss function while freezing factors of any other of the plurality of task-specific parameter matrices.
Claim 14
Claim 13
Patent cited above fails to teach the limitations at this claim, however, Alexandrovich teaches them at Paragraph 205.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teaching of Shen with the above combination of Bojarski and Cao as well as Alexandrovich to compensate for each error discovered during the learning process.
Claim 15
15. (Original) The computer-implemented method of claim 11, wherein the one or more prediction tasks comprise one or more of: semantic segmentation, depth estimation, object detection, instance segmentation, lane detection, surface normal Page 4 estimation, travelable area estimation, traffic sign recognition, natural language processing, classification, regression, emotion detection, intent detection, named entity recognition, or sentence boundary detection.
Claim 14
14. The method of claim 13, wherein the multiple different prediction tasks comprise semantic segmentation, depth estimation, object detection, instance segmentation, or surface normal estimation.
Claim 16
Claim 13
Patent cited above fails to teach the limitations at this claim, however, Bojarski teaches them at Page 7 Paragraph 3 Lines 1 to 3.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teaching of Shen with the above combination of Bojarski and Cao as well as Alexandrovich to achieve good performance.
Claim 17
Claim 13
Patent cited above fails to teach the limitations at this claim, however, Cao teaches them at Page 1 Lines 2 to 12.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teaching of Shen with the above combination of Bojarski and Cao to boost the performance of CNNs on many classical vision tasks.
Claim 18
18. (Original) A data processing system comprising means for performing the steps of a computer-implemented method according to claim 1.
Claim 7
7. A computing system for automatically controlling a vehicle, the computing system comprising one or more processors and memory storing one or more programs for execution by the one or more processors for, the one or more programs including instructions for:
receiving data generated using at least one sensor of the vehicle;
simultaneously performing multiple different prediction tasks on the data using a multi-task neural network, wherein the multi-task neural network comprises at least one shared parameter inference matrix comprising parameters shared between the multiple different prediction tasks, and the at least one shared parameter inference matrix was over-parameterized during training into at least one shared parameter matrix and multiple task-specific parameter matrices, each of the multiple task-specific parameter matrices being associated with a different one of the multiple different tasks; and
automatically controlling the vehicle based on results of the multiple different prediction tasks.
Claim 19
19. (Original) A computer program, a machine-readable storage medium, or a data carrier signal that comprises instructions, that upon execution on at least one of a data processing device or control unit comprising at least one processor, cause the at least one of the data processing device or control unit to perform the method according to claim 1.
Claim 7
7. A computing system for automatically controlling a vehicle, the computing system comprising one or more processors and memory storing one or more programs for execution by the one or more processors for, the one or more programs including instructions for:
receiving data generated using at least one sensor of the vehicle;
simultaneously performing multiple different prediction tasks on the data using a multi-task neural network, wherein the multi-task neural network comprises at least one shared parameter inference matrix comprising parameters shared between the multiple different prediction tasks, and the at least one shared parameter inference matrix was over-parameterized during training into at least one shared parameter matrix and multiple task-specific parameter matrices, each of the multiple task-specific parameter matrices being associated with a different one of the multiple different tasks; and
automatically controlling the vehicle based on results of the multiple different prediction tasks.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
The claim limitation being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112 is:
"means for performing the steps of a computer-implemented method according to claim 1" in claim 18.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
A review of the specification shows that the following appears to be the corresponding structure described in the specification for the 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph limitation:
Specification at paragraph [0096] recites: “For example, software 750 can include one or more programs for execution by one or more processor(s) 710 for performing one or more of the steps of method 300, method 500, and/or method 600”. Therefore, the corresponding structure of the means for recited in the claims is interpreted to be the processor 710, as this processor performs the method 500 which is the method of claim 1.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 19 is directed to non-statutory subject matter because the claim recites software per se which is not patent eligible under 101. The claimed system lacks physical components that leads one of ordinary skill to reasonably interpret the claimed subject matter as software or firmware. As such, one of ordinary skill could reasonably interpret the invention embodied by the claim as software or a software product, which is not a manufacture, machine, process or composition of matter and is not believed to be functionally and structurally interconnected with any (hardware) storage medium in such a manner as to enable it to act as a computer component and realize it's functionality. Thus, the claim fails to meet the requirements of 35 U.S.C. 101.
Claim 19 is further rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because the broadest reasonable interpretation of the “machine-readable storage medium” and the data carrier signal encompass signals per se. The specification discloses that “the machine-readable medium may be any medium, such as for example, read-only memory (ROM); random access memory (RAM); a universal serial bus (USB) stick; a compact disc (CD); a digital video disc (DVD); a data storage device; a hard disk; electrical, acoustical, optical, or other forms of propagated signals (e.g., digital signals, data carrier signal, carrier waves), or any other medium on which a program element as described above can be transmitted and/or stored” (see Paragraph 0043, lines 3-6). A claim whose BRI covers non-statutory embodiments embraces subject matter that is not eligible for patent protection and therefore is directed to non-statutory subject matter. See MPEP 2106.03(II). It is suggested that claim 19 be amended to recite a “non-transitory” computer readable medium to overcome this rejection and it is also suggested to remove a data carrier signal.
Accordingly, claim 19 fails to recite statutory subject matter under 35 U.S.C. 101.
Claims 1–2, 5-12, and 15-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1:
Step 1 - Is the claim directed to a process, machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim recites the abstract ideas:
performing one or more prediction tasks on the data using a neural network, wherein the neural network comprises at least one parameter tensor comprising at least one element, and the at least one parameter tensor was over-parameterized during training into a plurality of component tensors: - This claim is directed to a mathematical concept, as the process of performing a prediction tasks is a step of “determining” a variable or number using mathematical method (see MPEP 2106.04(a)(2) subsection C) . The use of the neural network is discussed next at Step 2A: Prong 2.
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
receiving data generated using at least one sensor of a machine: - This limitation is directed to mere data gathering and outputting. The courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) have recognized mere data gathering and outputting as insignificant extra-solution activity (see MPEP 2106.05(g)(3)) and therefore fails to integrate the exception into a practical application;
…using a neural network…- This limitation does not integrate a judicial exception into a practical application as the neural network is recited at a high level of generality, therefore this amounts to mere instructions to implement an abstract idea 2106.05(f).
controlling the machine based on results of the one or more prediction tasks: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of neural network (see MPEP 2106.05(h)).
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements:
receiving data generated using at least one sensor of a machine: - This limitation is directed to receiving or transmitting data over a network. The courts (as per Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362) have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) (se MPEP 2106.05(d) II);
…using a neural network… - This limitation does not amount to significantly more than the judicial exception as the neural network is recited at a high level of generality, therefore this amounts to mere instructions to implement an abstract idea 2106.05(f);
controlling the machine based on results of the one or more prediction tasks: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of neural network (see MPEP 2106.05(h)).
Regarding Claim 2:
Step 1 - Is the claim directed to a process, machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). Additionally, claim 2 recites the abstract ideas:
wherein the plurality of component tensors comprise an identical number of elements as the at least one parameter tensor and were compressed by element-wise addition after training to generate the at least one parameter tensor: - This claim is directed to a mathematical concept, as the process of compressing by element-wise addition to generate the parameter tensor is a step of organizing information and manipulating information through mathematical correlations (see MPEP 2106.04(a)(2)(I) subsection A). This claim is further directed to a mathematical concept because of the use of the element-wise addition to compress the elements of the component tensors.
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 5:
Step 1 - Is the claim directed to a process, machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1). Additionally, claim 5 recites the abstract idea:
carrying out a plurality of forward passes on the neural network to generate a plurality of predictions for each prediction task, wherein a subset of elements of the at least one parameter tensor is dropped out during each forward pass: - This claim is directed to a mathematical concept, as the process of generating a prediction tasks is a step of “determining” a variable or number using mathematical method (see MPEP 2106.04(a)(2) subsection C);
and determining at least one of a mean, a variance or entropy for each of the prediction tasks based on the plurality of predictions generated for each prediction task: - This claim is directed to a mathematical concept, as the process of determining a mean, a variance or entropy is a step of “determining” a variable or number using mathematical method (see MPEP 2106.04(a)(2) subsection C).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 6:
Step 1 - Is the claim directed to a process, machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
wherein the one or more prediction tasks comprise one or more of: semantic segmentation, depth estimation, object detection, instance segmentation, lane detection, surface normal estimation, travelable area estimation, traffic sign recognition, natural language processing, classification, regression, emotion detection, intent detection, named entity recognition, or sentence boundary detection: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of neural network (see MPEP 2106.05(h)).
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements:
wherein the one or more prediction tasks comprise one or more of: semantic segmentation, depth estimation, object detection, instance segmentation, lane detection, surface normal estimation, travelable area estimation, traffic sign recognition, natural language processing, classification, regression, emotion detection, intent detection, named entity recognition, or sentence boundary detection: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of neural network (see MPEP 2106.05(h)).
Regarding Claim 7:
Step 1 - Is the claim directed to a process, machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
wherein the machine corresponds to a mobile agent; and wherein controlling the machine based on results of the one or more prediction tasks comprises, by at least one processor, at least one of steering the mobile agent, braking the mobile agent, parking the mobile agent, or providing an alert to an operator of the mobile agent or a third party: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of neural network (see MPEP 2106.05(h)).
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements:
wherein the machine corresponds to a mobile agent; and wherein controlling the machine based on results of the one or more prediction tasks comprises, by at least one processor, at least one of steering the mobile agent, braking the mobile agent, parking the mobile agent, or providing an alert to an operator of the mobile agent or a third party: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of neural network (see MPEP 2106.05(h)).
Regarding Claim 8:
Step 1 - Is the claim directed to a process, machine, manufacture or composition of matter? - Yes, the claim is directed to a machine.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
A computing system for automatically controlling a machine, the computing system comprising one or more processors and memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for carrying out a computer-implemented method according to claim 1: - This limitation invokes a computer merely as a tool for performing an existing process [see MPEP 2106.05(f)(2)] and therefore fails to integrate the exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements:
A computing system for automatically controlling a machine, the computing system comprising one or more processors and memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for carrying out a computer-implemented method according to claim 1: This limitation invokes a computer merely as a tool for performing an existing process [see MPEP 2106.05(f)(2)] and therefore fails to amount to significantly more than the judicial exception.
Regarding Claim 9:
Step 1 - Is the claim directed to a process, machine, manufacture or composition of matter? - Yes, the claim is directed to a machine.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim is dependent on claim 8 and claim 8 depends on claim 1 which included an abstract idea (see rejection for claim 1).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
wherein the machine is a mobile agent, and the computing system is an embedded computing system of the mobile agent: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of autonomous vehicle (see MPEP 2106.05(h)).
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements:
wherein the machine is a mobile agent, and the computing system is an embedded computing system of the mobile agent: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of autonomous vehicle (see MPEP 2106.05(h)).
Regarding Claim 10:
Step 1 - Is the claim directed to a process, machine, manufacture or composition of matter? - Yes, the claim is directed to a machine.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim is dependent on claim 8 and claim 8 depends on claim 1 which included an abstract idea (see rejection for claim 1).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
A machine or mobile agent comprising at least one sensor and the computing system of claim 8: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of sensor (see MPEP 2106.05(h)).
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements:
A machine or mobile agent comprising at least one sensor and the computing system of claim 8: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of sensor (see MPEP 2106.05(h)).
Regarding Claim 11:
Step 1 - Is the claim directed to a process, machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? – No, the claim does not recite an abstract idea.
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
training a learning neural network on a plurality of training datasets, the neural network comprising at least one over-parameterized parameter tensor, the at least one over-parameterized tensor comprising a plurality of component tensors: - This limitation does not integrate a judicial exception into a practical application as the neural network is recited at a high level of generality, therefore this amounts to mere instructions to implement an abstract idea 2106.05(f);
generating a machine learned neural network for performing one or more prediction tasks on a dataset, the machine learned neural network comprising at least one parameter tensor that is a combination of the trained plurality of component tensors: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of neural network to perform prediction tasks on a combination of trained component tensors. (see MPEP 2106.05(h));
embedding the machine learned neural network into a computing system for the machine such that the computing system performs the one or more prediction tasks on the sensor data of the machine and controls the machine based on results of the one or more prediction tasks: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of autonomous vehicle (see MPEP 2106.05(h));
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements:
training a learning neural network on a plurality of training datasets, the neural network comprising at least one over-parameterized parameter tensor, the at least one over-parameterized tensor comprising a plurality of component tensors: - This limitation does not amount to significantly more than the judicial exception as the neural network is recited at a high level of generality, therefore this amounts to mere instructions to implement an abstract idea 2106.05(f);
generating a machine learned neural network for performing one or more prediction tasks on a dataset, the machine learned neural network comprising at least one parameter tensor that is a combination of the trained plurality of component tensors: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of neural network to perform prediction tasks on a combination of trained component tensors. (see MPEP 2106.05(h));
embedding the machine learned neural network into a computing system for the machine such that the computing system performs the one or more prediction tasks on the sensor data of the machine and controls the machine based on results of the one or more prediction tasks: - This limitation does no more than generally link a judicial exception to a particular technological environment as this limitation recites the use of judicial exception to the technology of autonomous vehicle (see MPEP 2106.05(h)).
Regarding claim 12, this claim is a method claim and is rejected on the same basis as claim 2 since they are analogous.
Regarding claim 15, this claim is a method claim and is rejected on the same basis as claim 6 since they are analogous.
Regarding claim 16, this claim is a method claim and is rejected on the same basis as claim 7 since they are analogous.
Regarding Claim 17:
Step 1 - Is the claim directed to a process, a process, machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? – No, the claim does not recite an abstract.
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
A data structure generated by the computer-implemented method of claim 11: - This limitation is directed to mere data gathering and outputting. The courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) have recognized mere data gathering and outputting as insignificant extra-solution activity (see MPEP 2106.05(g)(3)) and therefore fails to integrate the exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements:
A data structure generated by the computer-implemented method of claim 11: - This limitation is directed to receiving or transmitting data over a network. The courts (as per Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362) have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) (se MPEP 2106.05(d) II).
Regarding Claim 18:
Step 1 - Is the claim directed to a process, a process, machine, manufacture or composition of matter? - Yes, the claim is directed to a machine.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? – Yes, the claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
A data processing system comprising means for performing the steps of a computer-implemented method according to claim 1: - This limitation amounts to adding the words "apply it" (or an equivalent) with the judicial exception, as the processing system is considered as a generic computer that is used in its ordinary capacity to perform steps (see MPEP 2106.05(f)) and therefore fails to integrate the exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements:
A data processing system comprising means for performing the steps of a computer-implemented method according to claim 1: This limitation amounts to adding the words "apply it" (or an equivalent) with the judicial exception, as the processing system is considered as a generic computer that is used in its ordinary capacity to display data. (see MPEP 2106.05(f)) and therefore fails to amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 7-11, 16, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Bojarski et al (NPL: “End to End Learning for Self-Driving Cars”- hereinafter Bojarski) in view of Cao et al (NPL: "DO-Conv: Depthwise Over-parameterized Convolutional Layer"- hereinafter Cao).
Referring to Claim 1, Bojarski teaches a computer-implemented method for automatically controlling a machine, the method comprising:
receiving data generated using at least one sensor of a machine (see Bojarski Page 8 Paragraph 1 Sentence 2:”Top: subset of the camera image sent to the CNN” and further see Borjaski paragraph , 3rd and 4th paragraphs: The simulator sends the first frame of the chosen test video, adjusted for any departures from the ground truth, to the input of the trained CNN. The CNN then returns a steering command for that frame. The CNN steering commands as well as the recorded human-driver commands are fed into the dynamic model [8] of the vehicle to update the position and orientation of the simulated vehicle. The simulator then modifies the next frame in the test video so that the image appears as if the vehicle were at the position that resulted by following steering commands from the CNN. This new image is then fed to the CNN and the process repeats”. Examiner interprets the fact that the CNN receives video frame from the camera in order to output steering commands in an iterative manner as equivalent to the claimed “receiving data”, as the camera frames is interpreted as the sensor);
performing one or more prediction tasks on the data using a neural network (see Bojarski p 6, 3rd and 4th paragraphs: The simulator sends the first frame of the chosen test video, adjusted for any departures from the ground truth, to the input of the trained CNN. The CNN then returns a steering command for that frame. The CNN steering commands as well as the recorded human-driver commands are fed into the dynamic model [8] of the vehicle to update the position and orientation of the simulated vehicle. The simulator then modifies the next frame in the test video so that the image appears as if the vehicle were at the position that resulted by following steering commands from the CNN. This new image is then fed to the CNN and the process repeats”). Examiner interprets the CNN being used to return/output the steering commands as the claimed prediction tasks;
controlling the machine based on results of the one or more prediction tasks (see Borjaski Page 6 Lines 4 to 6: “The CNN steering commands as well as the recorded human-driver commands are fed into the dynamic model [8] of the vehicle to update the position and orientation of the simulated vehicle” further see Bojarski Page 7 Paragraph 3 Lines 1 to 3: “After a trained network has demonstrated good performance in the simulator, the network is loaded on the DRIVETM PX in our test car and taken out for a road test. For these tests we measure performance as the fraction of time during which the car performs autonomous steering”. Examiner interprets updating the position and orientation and ultimately the road-test to be equivalent as the claimed “controlling the machine”).
However, Bojarski fails to teach:
wherein the neural network comprises at least one parameter tensor comprising at least one element, and the at least one parameter tensor was over-parameterized during training into a plurality of component tensors.
Cao teaches, in an analogous system:
wherein the neural network comprises at least one parameter tensor comprising at least one element, and the at least one parameter tensor was over-parameterized during training into a plurality of component tensors (see Cao Page 1 Lines 2 to 12: “In this paper, we propose to augment a convolutional layer with an additional depthwise convolution, where each input channel is convolved with a different 2D kernel. The composition of the two convolutions constitutes an over-parameterization, since it adds learnable parameters, while the resulting linear operation can be expressed by a single convolution layer. We refer to this depth wise over-parameterized convolutional layer as DO-Conv. We show with extensive experiments that the mere replacement of conventional convolutional layers with DO-Conv layers boosts the performance of CNNs on many classical vision tasks, such as image classification, detection, and segmentation. Moreover, in the inference phase, the depthwise convolution is folded into the conventional convolution, reducing the computation to be exactly equivalent to that of a convolutional layer without over-parameterization”. Examiner interprets image classification, detection and segmentation to be equivalent as the claimed “prediction tasks”; convolutional layer and depthwise convolution are interpreted to be equivalent as the claimed “tensor”; each input channel is interpreted to be equivalent as the claimed “one element”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Bojarski with the above teachings of Cao by feeding the steering commands into the model to update the position and the orientation of the vehicle, as taught by Bojarski, and augmenting a convolutional layer with an additional depthwise convolution, as taught by Cao. The modification would have been obvious because one of ordinary skill in the art would be motivated to boost performance in computer vision tasks (as suggested by Cao at page 1 lines 7 to 10: “We show with extensive experiments that the mere replacement of conventional convolutional layers with DO-Conv layers boosts the performance of CNNs on many classical vision tasks, such as image classification, detection, and segmentation”).
Referring to Claim 7, the combination of Bojarski and Cao teaches the method of claim 1:
wherein the machine corresponds to a mobile agent (see Bojarski Page 7 Paragraph 3 Lines 1 to 3: “After a trained network has demonstrated good performance in the simulator, the network is loaded on the DRIVETM PX in our test car and taken out for a road test. For these tests we measure performance as the fraction of time during which the car performs autonomous steering”. Examiner interprets the test car to be equivalent as the claimed “mobile agent”);
wherein controlling the machine based on results of the one or more prediction tasks comprises, by at least one processor, at least one of steering the mobile agent, braking the mobile agent, parking the mobile agent, or providing an alert to an operator of the mobile agent or a third party (see Bojarski Page 7 Paragraph 3 Lines 1 to 3: “After a trained network has demonstrated good performance in the simulator, the network is loaded on the DRIVETM PX in our test car and taken out for a road test. For these tests we measure performance as the fraction of time during which the car performs autonomous steering”. Examiner interprets the car performs autonomous steering to be equivalent as the claimed “steering the mobile agent”).
Referring to Claim 8, the combination of Bojarski and Cao teaches a computing system:
for automatically controlling a machine, the computing system comprising one or more processors and memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for carrying out a computer-implemented method according to claim 1 (see Bojarski Page 7, 3rd Paragraph Lines 1 to 2: “After a trained network has demonstrated good performance in the simulator, the network is loaded on the DRIVETM PX in our test car and taken out for a road test.”. Examiner interprets the DRIVETM PX to be equivalent as the claimed “computing system comprising one or more processors and memory storing”).
Referring to Claim 9, the combination of Bojarski and Cao teaches the computing system of claim 8:
wherein the machine is a mobile agent, and the computing system is an embedded computing system of the mobile agent (see Bojarski Page 7 Paragraph 3 Lines 1 to 2: “After a trained network has demonstrated good performance in the simulator, the network is loaded on the DRIVETM PX in our test car and taken out for a road test”. Examiner interprets the DRIVETM PX and the test car to be equivalent as the claimed “computing system” and “mobile agent” respectively).
Referring to Claim 10, the combination of Bojarski and Cao teaches:
a machine or mobile agent comprising at least one sensor and the computing system of claim 8 (see Bojarski Page 4 Paragraph 2 Lines 1 to 2: “Data was acquired using either our drive-by-wire test vehicle, which is a 2016 Lincoln MKZ, or using a 2013 Ford Focus with cameras placed in similar positions to those in the Lincoln. The system has no dependencies on any particular vehicle make or model”. Examiner interprets the test vehicle, the cameras and the system to be equivalent as the claimed “machine or mobile agent”, “sensor” and “computing system” respectively).
Referring to Claim 11, Bojarski teaches a computer-implemented method for generating a machine learned neural network that can perform one or more prediction tasks based on data of sensors of a machine for automatically controlling the machine, the computer-implemented method comprising:
embedding the machine learned neural network into a computing system for the machine such that the computing system performs the one or more prediction tasks on the sensor data of the machine and controls the machine based on results of the one or more prediction tasks (see Borjaski Page 6 Lines 4 to 6: “The CNN steering commands as well as the recorded human-driver commands are fed into the dynamic model [8] of the vehicle to update the position and orientation of the simulated vehicle” further see Bojarski Page 7 Paragraph 3 Lines 1 to 3: “After a trained network has demonstrated good performance in the simulator, the network is loaded on the DRIVETM PX in our test car and taken out for a road test. For these tests we measure performance as the fraction of time during which the car performs autonomous steering” . Examiner interprets loading the network on the DRIVETM PX to be equivalent as the claimed “embedding the machine learned neural network into a computing system”; updating the position and orientation and ultimately the road-test is interpreted to be equivalent as the claimed “performs the one or more prediction tasks on the sensor data of the machine and controls the machine based on results of the one or more prediction tasks”).
However, Bojarski fails to teach:
training a learning neural network on a plurality of training datasets, the neural network comprising at least one over-parameterized parameter tensor, the at least one over-parameterized tensor comprising a plurality of component tensors; and
generating a machine learned neural network for performing one or more prediction tasks on a dataset, the machine learned neural network comprising at least one parameter tensor that is a combination of the trained plurality of component tensors.
Cao teaches, in an analogous system:
training a learning neural network on a plurality of training datasets, the neural network comprising at least one over-parameterized parameter tensor, the at least one over-parameterized tensor comprising a plurality of component tensors (see Cao p 7, 1st paragraph: “We also conducted semantic segmentation experiments on the PASCALVOC[11] and the Cityscapes datasets[8] with GluonCV, following the same comparison protocol as that used for the ImageNet classification task. Differently from image classification, the training of a CNN model for image segmentation often has two stages: the “Backbone” and “Segmentation”. The first stage pre-trains a backbone model on the ImageNet classification task, and the second stage fine-tunes the backbone for the segmentation task. DO-Conv can be used in one or both of the stages. We use Deeplabv3[5] with ResNet-50 and ResNet100 as the backbones for Cityscapes and PASCALVOC datasets, respectively, and summarize the results in Table4. Note that the delta in both the second and third rows are relative to the first row. We can observe that DO-Conv consistently boosts the performance, either when used only in the second stage, or in both stages”. Examiner interprets the training of the CNN on ImageNet (first stage) and on ResNet-50 as well as on ResNet-100 (second stage) to be equivalent as the claimed “training a learning neural network on a plurality of training datasets”, and further (see Cao p 3, 5th paragraph : “Depthwise over-parameterized convolutional layer (DO-Conv) is a composition of a depthwise convolution” DO-Conv is the over-parameterized tensor interpreted to be equivalent as the claimed “one over-parameterized tensor comprising a plurality of component tensors”);
generating a machine learned neural network for performing one or more prediction tasks on a dataset, the machine learned neural network comprising at least one parameter tensor that is a combination of the trained plurality of component tensors (see Cao Page 1 Lines 2 to 12: “In this paper, we propose to augment a convolutional layer with an additional depthwise convolution, where each input channel is convolved with a different 2D kernel. The composition of the two convolutions constitutes an over-parameterization, since it adds learnable parameters, while the resulting linear operation can be expressed by a single convolution layer. We refer to this depth wise over-parameterized convolutional layer as DO-Conv. We show with extensive experiments that the mere replacement of conventional convolutional layers with DO-Conv layers boosts the performance of CNNs on many classical vision tasks, such as image classification, detection, and segmentation. Moreover, in the inference phase, the depthwise convolution is folded into the conventional convolution, reducing the computation to be exactly equivalent to that of a convolutional layer without over-parameterization”. Examiner interprets the resulting linear operation being expressed by a single convolution layer to be equivalent as the claimed “ the machine learned neural network comprising at least one parameter tensor that is a combination of the trained plurality of component tensors”; image classification, detection and segmentation are interpreted to be equivalent as the claimed “prediction tasks).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Bojarski with the above teachings of Cao by feeding the steering commands into the model to update the position and the orientation of the vehicle, as taught by Bojarski, and augmenting a convolutional layer with an additional depthwise convolution, as taught by Cao. The modification would have been obvious because one of ordinary skill in the art would be motivated to boost performance in computer vision tasks (as suggested by Cao at page 1 lines 7 to 10: “We show with extensive experiments that the mere replacement of conventional convolutional layers with DO-Conv layers boosts the performance of CNNs on many classical vision tasks, such as image classification, detection, and segmentation”).
Referring to dependent Claim 16, this claim is rejected on the same basis as dependent claim 7 since they are analogous claims.
Referring to Claim 19, Bojarski teaches:
a computer program, a machine-readable storage medium, or a data carrier signal that comprises instructions, that upon execution on at least one of a data processing device or control unit comprising at least one processor, cause the at least one of the data processing device or control unit to perform the method according to claim 1 (see Bojarski Page 4 Paragraph 2 Lines 1 to 2: “Data was acquired using either our drive-by-wire test vehicle, which is a 2016 Lincoln MKZ, or using a 2013 Ford Focus with cameras placed in similar positions to those in the Lincoln. The system has no dependencies on any particular vehicle make or model”. Examiner interprets the test vehicle, the cameras and the system to be equivalent as the claimed “machine-readable storage medium or data carrier signal”).
Claims 2-6, 12-15, 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Bojarski et al (NPL: “End to End Learning for Self-Driving Cars”- hereinafter Bojarski) in view of Cao et al(NPL: "DO-Conv: Depthwise Over-parameterized Convolutional Layer"- hereinafter Cao) and in further view of Alexandrovich et al (CN 116965029A - hereinafter Alexandrovich).
Referring to Claim 2, the combination of Bojarski and Cao teaches the method of claim 1, however, fails to teach:
wherein the plurality of component tensors comprise an identical number of elements as the at least one parameter tensor and were compressed by element-wise addition after training to generate the at least one parameter tensor.
Alexandrovich teaches, in analogous system:
wherein the plurality of component tensors comprise an identical number of elements as the at least one parameter tensor and were compressed by element-wise addition after training to generate the at least one parameter tensor (see Alexandrovich paragraph 329: “Referring to Figures 15 and 16, the updatable sublayer corresponds to the incremental part (i.e., the incremental layer), and the pre-configured sublayer corresponds to the original part (i.e., the original layer). Furthermore, the same input to the updatable layer is provided to both the updatable sublayer and the preconfigured sublayer; the output of the updatable layer is the sum of the outputs of the updatable sublayer and the preconfigured sublayer, which is 1730. Providing the same input to both sub-layers means that the pre-configured layer of the updatable sub-layer and the updatable layer are in parallel, as shown in Figures 15 and 16. In addition, the corresponding outputs of the sub-layers are summed (added together). In one example, you can perform an element-wise sum on all elements of the sublevel”. Examiner interprets sub-layers to be equivalent as the claimed “component tensors” and the element-wise summation of the outputs is being interpreted as the claimed “compressed by element-wise addition”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Bojarski and Cao with the above teachings of Alexandrovich by controlling a machine based on prediction tasks performed on neural networks that is over-parameterized, as taught by Bojarski and Cao, and performing element-wise summation on all outputs of sub-layers, as taught by Alexandrovich. The modification would have been obvious because one of ordinary skill in the art would be motivated to accelerate the training (as suggested by Alexandrovich at paragraph 330: “Therefore, the image decoding is accelerated because only the weight changes are updated, while pre-configured weights are used as a reference”).
Referring to Claim 3, the combination of Bojarski and Cao teaches the method of claim 1, however, fails to teach:
wherein during training of the neural network, a subset of the plurality of component tensors is trained at each training epoch by updating elements of the subset of the plurality of component tensors while freezing elements of any other component tensors
Alexandrovich teaches, in analogous system:
wherein during training of the neural network, a subset of the plurality of component tensors is trained at each training epoch by updating elements of the subset of the plurality of component tensors while freezing elements of any other component tensors (see Alexandrovich paragraph 11: “Using sub-layers of the updatable layer and training the weights of the updatable sub-layers while maintaining pre-configured weights helps to train the updatable layer based on weight changes relative to the pre-configured weights”. Examiner interprets maintaining pre-configured sub-layers weights to be equivalent as the claimed “freezing elements of any other component tensors”);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Bojarski and Cao with the above teachings of Alexandrovich by controlling a machine based on prediction tasks performed on neural networks that is over-parameterized, as taught by Bojarski and Cao, and updating some sublayers and freezing the rest, as taught by Alexandrovich. The modification would have been obvious because one of ordinary skill in the art would be motivated to accelerate the training (as suggested by Alexandrovich at paragraph 11: “Therefore, NN training can be accelerated by retraining and updating only a subset (one or more, but not all) of all layers of the neural network, i.e., the one or more updatable layers with updatable parameters, thereby reducing the time spent on NN training, since pre-configured weights are used as a reference for the training. In other words, the convergence speed of the NN training can be accelerated”).
Referring to Claim 4, the combination of Bojarski and Cao teaches the method of claim 3, however, fails to teach:
the subset of the plurality of component tensors comprises one component tensor;
the subset of the plurality of component tensors is selected randomly, wherein the selection is based on a probability of dropout associated with each of the plurality of component tensors.
Alexandrovich teaches, in analogous system:
the subset of the plurality of component tensors comprises one component tensor (see Alexandrovich Paragraph 11: “In one exemplary implementation, the updatable layer includes an updatable sublayer with weights of the one or more updatable parameters and a preconfigured sublayer with weights of the one or more preconfigured parameters”. Examiner interprets sublayer to be equivalent as the claimed “component tensor”);
the subset of the plurality of component tensors is selected randomly, wherein the selection is based on a probability of dropout associated with each of the plurality of component tensors (see Alexandrovich Paragraph 205: “Weight updates can be accomplished using stochastic gradient descent (SGD) or other methods”. Examiner interprets the weight updates using stochastic gradient descent to be equivalent as the claimed “selection is based on a probability of dropout associated with each of the plurality of component tensors”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Bojarski and Cao with the above teachings of Alexandrovich by controlling a machine based on prediction tasks performed on neural networks that is over-parameterized, as taught by Bojarski and Cao, and updating weights using stochastic gradient descent, as taught by Alexandrovich. The modification would have been obvious because one of ordinary skill in the art would be motivated to compensate for each error discovered during the learning process (as suggested by Alexandrovich at paragraph 205: “Backpropagation: is a method that adjusts weights to compensate for each error discovered during the learning process. Technically, backpropagation calculates the gradient (derivative) of the cost function associated with a given state with respect to the weights. Weight updates can be accomplished using stochastic gradient descent (SGD) or other methods”).
Referring to Claim 5, the combination of Bojarski and Cao teaches the method of claim 1 wherein performing one or more prediction tasks on the data using a neural network comprises, however, fails to teach:
carrying out a plurality of forward passes on the neural network to generate a plurality of predictions for each prediction task;
determining at least one of a mean, a variance or entropy for each of the prediction tasks based on the plurality of predictions generated for each prediction task.
Alexandrovich teaches, in analogous system:
carrying out a plurality of forward passes on the neural network to generate a plurality of predictions for each prediction task, wherein a subset of elements of the at least one parameter tensor is dropped out during each forward pass (see Alexandrovich Paragraph 125: “In other words, the segmentation unit 262 can be used to divide the block 203 into smaller segments or sub-blocks (forming blocks again), for example, using quad-tree (QT) segmentation, binary-tree (BT) segmentation, or triple-tree (TT) segmentation, or any combination thereof, iteratively, and for example, to predict each segment or sub-block, wherein the mode selection includes selecting the tree structure of the segment block 203 and applying the prediction mode to each segment or sub-block”. Examiner interprets dividing the block into smaller sub-blocks in iterative manner to be equivalent as the “carrying out a plurality of forward passes on the neural network to generate a plurality of predictions for each prediction task” and selecting the structure of the block and applying the prediction mode to be equivalent as the claimed “wherein a subset of elements of the at least one parameter tensor is dropped out during each forward pass”);
determining at least one of a mean, a variance or entropy for each of the prediction tasks based on the plurality of predictions generated for each prediction task (see Alexandrovich Paragraph 132: “The intra-prediction mode set may include 35 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in HEVC, or may include 67 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in VVC”. Examiner interprets the different intra-prediction modes to be equivalent as the claimed “a mean, a variance or entropy for each of the prediction tasks based on the plurality of predictions generated for each prediction task”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Bojarski and Cao with the above teachings of Alexandrovich by controlling a machine based on prediction tasks performed on neural networks that is over-parameterized, as taught by Bojarski and Cao, and carrying out a plurality of forward passes to generate a plurality of predictions for each prediction task, as taught by Alexandrovich. The modification would have been obvious because one of ordinary skill in the art would be motivated to achieve better compression in transmission or storage (as suggested by Alexandrovich at paragraph 124: “In one embodiment, the mode selection unit 260 may be used to select a segmentation and prediction mode (e.g., from prediction modes supported or available by the mode selection unit 260), which provides the best match or the minimum residual (minimum residual means better compression in transmission or storage), or provides the minimum indication overhead (minimum indication overhead means better compression in transmission or storage), or considers or balances both”).
Referring to Claim 6, the combination of Bojarski and Cao teaches the method of claim 1, however, fails to teach:
wherein the one or more prediction tasks comprise one or more of: semantic segmentation, depth estimation, object detection, instance segmentation, lane detection, surface normal estimation, travelable area estimation, traffic sign recognition, natural language processing, classification, regression, emotion detection, intent detection, named entity recognition, or sentence boundary detection.
Alexandrovich teaches, in analogous system:
wherein the one or more prediction tasks comprise one or more of: semantic segmentation, depth estimation, object detection, instance segmentation, lane detection, surface normal estimation, travelable area estimation, traffic sign recognition, natural language processing, classification, regression, emotion detection, intent detection, named entity recognition, or sentence boundary detection (see Alexandrovich Paragraph 200: “Today, CNNs are the most commonly used method for computer vision (CV) tasks, such as classification, Face ID, person re-identification, car brand recognition, object detection, semantic and instance segmentation, image/video augmentation, and image/video super-resolution”. Examiner interprets the computer vision (CV) tasks to be equivalent as the claimed “prediction tasks”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Bojarski and Cao with the above teachings of Alexandrovich by controlling a machine based on prediction tasks performed on neural networks that is over-parameterized, as taught by Bojarski and Cao, and carrying out a plurality of forward passes to generate a plurality of predictions for each prediction task, as taught by Alexandrovich. The modification would have been obvious because one of ordinary skill in the art would be motivated to mitigate the challenges posed by MLP architectures (as suggested by Alexandrovich at paragraph 185: “CNN models mitigate the challenges posed by MLP architectures by leveraging the strong spatial local correlations present in natural images”).
Referring to dependent Claim 12, this claim is rejected on the same basis as dependent claim 2 since they are analogous claims.
Referring to dependent Claim 13, this claim is rejected on the same basis as dependent claim 3 since they are analogous claims.
Referring to dependent Claim 14, this claim is rejected on the same basis as dependent claim 4 since they are analogous claims.
Referring to dependent Claim 15, this claim is rejected on the same basis as dependent claim 6 since they are analogous claims.
Referring to Claim 17, the combination of Bojarski and Cao teaches:
a data structure generated by the computer-implemented method of claim 11 (see Cao Page 1 Lines 2 to 12: “In this paper, we propose to augment a convolutional layer with an additional depthwise convolution, where each input channel is convolved with a different 2D kernel. The composition of the two convolutions constitutes an over-parameterization, since it adds learnable parameters, while the resulting linear operation can be expressed by a single convolution layer. We refer to this depth wise over-parameterized convolutional layer as DO-Conv. We show with extensive experiments that the mere replacement of conventional convolutional layers with DO-Conv layers boosts the performance of CNNs on many classical vision tasks, such as image classification, detection, and segmentation. Moreover, in the inference phase, the depthwise convolution is folded into the conventional convolution, reducing the computation to be exactly equivalent to that of a convolutional layer without over-parameterization”. Examiner interprets the single convolution layer to be equivalent as the claimed “data structure generated”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Bojarski with the above teachings of Cao by feeding the steering commands into the model to update the position and the orientation of the vehicle, as taught by Bojarski, and augmenting a convolutional layer with an additional depthwise convolution, as taught by Cao. The modification would have been obvious because one of ordinary skill in the art would be motivated to boost performance of CNNs on many classical vision tasks (as suggested by Cao at page 1 lines 2 to 12: “In this paper, we propose to augment a convolutional layer with an additional depthwise convolution, where each input channel is convolved with a different 2D kernel. The composition of the two convolutions constitutes an over-parameterization, since it adds learnable parameters, while the resulting linear operation can be expressed by a single convolution layer. We refer to this depth wise over-parameterized convolutional layer as DO-Conv. We show with extensive experiments that the mere replacement of conventional convolutional layers with DO-Conv layers boosts the performance of CNNs on many classical vision tasks, such as image classification, detection, and segmentation. Moreover, in the inference phase, the depthwise convolution is folded into the conventional convolution, reducing the computation to be exactly equivalent to that of a convolutional layer without over-parameterization”).
Referring to Claim 18, the combination of Bojarski and Cao teaches:
a data processing system comprising means for performing the steps of a computer-implemented method according to claim 1 (see Bojarski Page 7, 3rd Paragraph Lines 1 to 2: “After a trained network has demonstrated good performance in the simulator, the network is loaded on the DRIVETM PX in our test car and taken out for a road test.”. Examiner interprets the DRIVETM PX to be equivalent as the claimed “data processing system”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AWADAGBE G HOUNTON whose telephone number is (571)270-0670. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AWADAGBE G HOUNTON/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights