DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This communication is in response to Application 19/052,117 filed on 02/12/2025. Claims 1-20 are currently pending and examined below.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/24/2025, 02/25/2025, 06/16/2025 and 10/21/2025 have been considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: a first sensor configured to, second sensor configured to, a converter configured to in claims 9-10.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The determination of whether a claim recites patent ineligible subject matter is a two-step inquiry.
Step 1: the claim does not fall within one of the four statutory categories of invention (process, machine, manufacture or composition of matter), See MPEP 2106.03, or
Step 2: the claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis: See MPEP 2106.04
Step 2A (Prong 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP 2106.04(II)(A)(1)
Step 2A (Prong 2): Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP 2106.04(II)(A)(2)
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP 2106.05
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1. A system comprising:
a sensor [generic computer/sensor component]; and
a memory device configured to [generic computer component],and
generate inference results using data collected from the sensor [generic computer/sensor component] as input to a first portion of an artificial neural network (ANN) [mental process/mathematical concept]; and
receive a second portion of the ANN from a host system, wherein the host system sends the second portion [insignificant post-solution activity (receiving data)] based on an evaluation of the inference results [mental process].
Claim 9. A system comprising:
a first sensor configured to generate image data [generic computer/sensor component];
a second sensor configured to generate non-image data [generic computer/sensor component]; and
a controller configured [generic computer component] to generate inference results using the image data and non-image data [mental process/mathematical concept].
Claim 14. A system comprising:
a memory device configured to store inference results [generic computer component]; and
at least one controller configured to [generic computer component]:
send the inference results to a host system for processing using a first portion of an artificial neural network (ANN) [insignificant extra solution activity (transmitting data)]; and
receive a second portion of the ANN from the host system [insignificant post-solution activity (receiving data)], wherein the host system sends the second portion [insignificant extra solution activity (transmitting data)] in response to determining data is being dropped due to communication bandwidth limitations [mental process/step].
101 Analysis – Step 1: Statutory Category – Yes
Claims 1, 9 and 14 recite a system. The claim falls within one of the four statutory categories. See MPEP 2106.03
Step 2A, Prong one evaluation: Judicial exception – Yes- Mental processes
In Step 2A, Prong one of the 2019 Patent Eligibility Guidance (PEG), a claim is to be analyzed to determine whether it recites subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) mental processes, and/or c) certain methods of organizing human activity. See MPEP 2106(A)(II)(1) and MPEP 2106.04(a)-(c)
The office submits that the foregoing bolded limitation(s) constitutes judicial exceptions in terms of “mental processes and mathematical concept” because under its broadest reasonable interpretation, the limitations can be “performed in the human mind, or by a human using a pen and paper.” See MPEP 2106.04(a)(2)(III).
The claim recites the limitations of generate inference results using data collected as input to a first portion of an artificial neural network (ANN), based on an evaluation of the inference results, generate inference results using the image data and non-image data, and determining data is being dropped due to communication bandwidth limitations. These limitations, as drafted, are simple processes that, under its broadest reasonable interpretation, covers performance of the limitation in the mind and mathematical concept but for the recitation of “a sensor, a memory, a first sensor, a second sensor, a controller ” That is, other than reciting “a sensor, a memory, a first sensor, a second sensor, a controller” nothing in the claim elements precludes the step from practically being performed in the mind. For example, but for the “a sensor, a memory, a first sensor, a second sensor, a controller” That is, other than reciting “display device, processing circuitry, remote assistance screen and monitoring images”, the claim encompasses a person looking at data collected and forming a simple judgement. The mere nominal recitation of sensor does not take the claim limitations out of the mental process grouping.
Thus, the claim recites a mental process.
Step 2A, Prong two evaluation: Practical Application - No
In Step 2A, Prong two of the 2019 PEG, a claim is to be evaluated whether, as a whole, it integrates the recited judicial exception into a practical application. As noted in MPEP 2106.04(d), it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception. The courts have indicated that additional elements such as: merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
The Office submits that the foregoing underlined limitation(s) recite additional elements that do not integrate the recited judicial exception into a practical application.
The claim recites additional elements or steps of display device, processing circuitry, remote assistance screen and monitoring images. In particular, the “a sensor, a memory, a first sensor, a second sensor, a controller” limitation is recited at a high level of generality (i.e. generic processor performing a generic computer function) such that it amounts to no more than mere instructions to “apply” the exception using a generic computer component
Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
Step 2B evaluation: Inventive concept - No
In Step 2B of the 2019 PEG, a claim is to be evaluated as to whether the claim, as a whole, amounts to significantly more than the recited exception, i.e. whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP 2106.05.
As discussed with respect to Step 2A Prong Twp, the additional elements in the claim amount to no more than mere instructions to apply the exception using a generic computer component. The same analysis applies here in 2B, i.e. mere instructions to apply an exception on a generic computer cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B. See MPEP 2106.05(f).
Under the 2019 PEG, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B. Here, the a sensor, a memory, a first sensor, a second sensor, a controller were considered to be insignificant extra-solution activity in Step 2A, and thus they are re-evaluated in Step 2B to determine if they are more than what is well-understood, routine, conventional activity in the field.
The specification recites that generic computer (See ¶210-214 of applicant’s specification). MPEP 2106.05(d)(II). Thus, the claim is ineligible.-
Dependent claim(s) 2-8, 10-13 and 15-20 do not recite any further limitations that cause the claim(s) to be patent eligible. Rather, the limitations of dependent claims are directed toward additional aspects of the judicial exception and/or well-understood, routine and conventional additional elements that do not integrate the judicial exception into a practical application. Therefore, dependent claims 2-8, 10-13 and 15-20 are not patent eligible under the same rationale as provided for in the rejection of claims 1, 9 and 14
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-11 and 13-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-13 and 17 of U.S. Patent No. 12246736. Although the claims at issue are not identical, they are not patentably distinct from each other because Issued claim 1 of U.S. Patent No. 12246736 recites a system having an image and non-image sensors, controller, a memory device storing a portion of an artificial neural network (ANN), an inference engine configured to generate inference results using sensor-derived data, a host system storing another portion of the ANN, transmission of the inference results to the host system for further ANN processing, and receipt of an additional ANN portion from the host system. Issued claim 1 further recites that the host system sends the additional ANN portion in response to evaluating data using the host side ANN portion and determining that images frames are being dropped due to communication bandwidth limitations. Issued claim 6 further recites sending the additional ANN portion in response to evaluating inference results received from the sensing device. Pending claims 1, 2 and 13 correspond to issued claims 1 and 6. The pending claims differ principally by broadly reciting a sensor, assigning receipt of the host transmitted ANN portion to the memory device or controller, and using different ordinal designations for the respective ANN portions. These differences do not render the pending claims patentably distinct. The ordinal designations merely distinguish the ANN portions within the respective claims and do not require structural or functional differences. Further, configuring the memory device or its controller to receive the ANN portion transmitted through the host interface would have been a predictable allocation of the disclosed host interface and memory storage functions.
Pending claims 3-8 correspond respectively to issued claims 8-13. Issued claim 8 recites a sonar sensor; issued claim 9 recites a package encapsulating the image sensor and memory device; issued claim 10 recites the sensor and memory device formed on respective integrated circuit does and stacked to form a three-dimensional integrated circuit; issued claim 11 recites image classification inference results; issued claim 12 recites a control for steering, braking, or acceleration of a vehicle based on the inference results; and issued claim 13 recites communication with the host system according to a serial communication protocol. The pending claims recite the same dependent features in connection with the broader system of pending claim 1 and therefore do not define patentably distinct subject matter.
Pending claim 9-11 correspond to issued claims 1 and 11. Issued claim 1 recites a first image sensor generating image data, a second sensor generating non-image sound data, and generation of inference results using both the image data and the sensor derived data as ANN inputs. Issued claim 11 further recites that the inference results include object detection. Pending claims 9-11 differ principally by broadly assigning the inference operation to a controller, broadly reciting the use of image and non-image data, and reciting identification of an object. These differences are not patentably distinct because implementing the inference engine functionality in a controller is a predictable arrangement of known processing components; the broad recitation of “using” the non-image data encompasses use following preprocessing or conversion; and identifying an object is an expected result of the expressly claim object detection operation.
Pending claim 14-20 correspond to issued claims 1-6 and 17. Issued claims 1, 4 and 5 recite storing inference results in a memory device, sending inference results to a host system, host side processing of the inference results, host issued write commands, and controller execution of read and write commands. Issued claim 17 recites sending inference results to a computing system for processing using a host stored ANN portion and receiving another ANN portion from the computing system in response to determining that image frames are being dropped due to communication bandwidth
Limitations.
Issued claim 2 recites a camera providing an image system, corresponding to pending claim 15. Issued claim 3 recites that the camera generates data at a rate exceeding the communication bandwidth and that the host accesses the inference results by sending commands to the host inference, corresponding to pending claims 16 and 17. Issued claims 1 and 4 recite host issued write commands associated with storage operations, corresponding to pending claim 18. Issued claim 5 recites controller execution of a read command requesting access to stored inference results, corresponding to pending claim 19. Issued claim 6 recites that the host sends the additional ANN portion in response to evaluating the inference results, corresponding to pending claim 20.
Pending claims 14-20 differ principally by reciting the subject matter as a broadly defined system, assigning receipt of the host transmitted ANN portion to the controller, broadly referring to dropped image frames as “data” and changing the ordinal designations of the ANN portions. These differences do not impart a patentable distinction. The term “data” encompasses the expressly claimed image frames; the ordinal labels do not require structurally different ANN portions; and implementing the operations of issued claim 17 using the controller, memory device, and host interface architecture of issued claims 1, 4 and 5 would have been a predictable implementation of the patented system.
Accordingly, pending claims 1-11 and 13-20 are anticipated by, or would have been obvious variations of the subject matter of issued claims 1-13 and 17 of U.S. Patent No. 12246736 and are not patentably distinct.
Claim 12 is rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 and 11 of U.S. Patent No. 12246736 in view of Kwon et al., US, 20200218979A1.
Issued claims 1 and 11 of U.S. Patent No. 12246736 recite generating inference results from sensor derived data and using the inference results for object detection, but do not expressly recite using the inference results to determine a distance to an object.
Kwon discloses applying sensor or image data to a machine learning model to generate object detection and object distance outputs. Kwon further discloses computing depth or distance values corresponding to objects depicted in an image and associating the depth values with the detected objects to determine the distance to an object.
It would have been obvious to configure the object detection system of issued claims 1 and 11 to additionally determine the distance to a detected object, as taught Kwon, in order to provide spital information regarding the detected object for use in navigation, path planning, obstacle avoidance, and vehicle control operations. Therefore, the additional distance determination feature of pending claim 12 does not render claim 12 patentably distinct from the patented claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 5-6 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Bigioi et al., US 20190065410A1 in view of Dunne et al., US 20200272899A1, hereinafter referred to as Bigioi and Dunne, respectively.
Regarding claim 1, Bigioi discloses system comprising:
a sensor (For applications, particularly those concerned with the environment external to a vehicle, still further types of image sensors could also be employed for example, a LIDAR image sensor or RADAR image sensor indeed the image sensors could also be supplemented with signals from microwave sensors – See at least ¶78); and
a memory device configured to (Such peripheral storage devices typically comprise a controller portion including a CPU accessing program memory and volatile memory across a parallel system bus – See at least ¶30):
generate inference results using data collected from the sensor as input to a first portion of an artificial neural network (ANN) (The host controller in conjunction with the device CPU and PCNN CPU loads input image(s) via the peripheral system memory into shared memory – See at least ¶54. Configuring a PCNN according to a neural network program and weights, processing the input images according to the neural network program and storing the resulting output in memory (i.e. generate inference results) – See at least ¶54-57. Distributing neural network layers (first portion) among multiple PCNN cores and clusters and communicating the output of one CNN – See at least ¶62-63. The input image data is generated by image sensors connected to the neural network processing device – See at least ¶78); and
receive a second portion of the ANN from a host system (The file system acts as a carrier for input data for the CNNs of the cluster as well as results from the CNNs. The file system is also used by the host to transfer the network data (layer configuration, type, weights, instructions) to the PCNN cluster -See at least ¶29. Writes the network configuration into the CNN network file or subdirectory through a file system API write command – See at least ¶35. The host controller in conjunction with the device CPU and PCNN CPU loads neural network configuration and weights from memory (not shown) into shared memory – See at least ¶53. Examiner notes the host supplied layer configuration and associated weights defining one or more neural network layers correspond to the claimed second portion of the ANN).
Bigioi fails to disclose wherein the host system sends the second portion based on an evaluation of the inference results.
However, Dunne teaches wherein the host system sends the second portion based on an evaluation of the inference results (Centralized site/device may generate an updated neural network in operation block by using the datasets and inference results received from the new edge device to retrain the neural network (originally trained in operation block). In operation, the centralized site/device may send the updated neural network information (an updated neural network, updated models, updated weights, etc.) to the edge device – See at least ¶96. In operation block, the edge device may use the information received from the centralized site/device (e.g., portions of the updated neural network, updated models, updated weights, etc.) to update the previously-received neural network (received in operation block) and execute the updated neural network – See at least ¶97. Examiner notes, the updated neural network portion sent by the centralized site/device corresponds to the claimed second portion sent based on evaluation of the inference results).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bigioi and include the feature of wherein the host system sends the second portion based on an evaluation of the inference results, as taught by Dunne, in order to update a locally executed neural network based on actual inference performance while reducing an amount of neural network data transmitted to a peripheral device.
Regarding claim 5, Bigioi, as modified, discloses wherein: the sensor is formed on a first integrated circuit die; the memory device is formed on at least one second integrated circuit die; and the first integrated circuit die and the at least one second integrated circuit die are stacked to form a three-dimensional integrated circuit (Implementing the neural processing architecture using stacked dies and identifies three dimensional wafer-to-wafer, die-to-wafer, and die-to-die boding. Implementing the controller and neural processing logic on one die, shared memory on one or more additional dies, and stacking a CCD sensor or other sensor die onto the processor/memory stack. The sensor and memory device are formed on respective integrated circuit dies that are stacked to form a three-dimensional integrated circuit – See at least ¶68, 73 and 75).
Regarding claim 6, Bigioi, as modified, discloses wherein the inference results include image classification (Neural network outputs including Face Detection & Tracking, Face Features Detection, Face & Iris Recognition, Eye Opening & Gaze, Fatigue & Drowsiness, Age, Gender, Emotions, Vital Sign Detection, Pose Invariant Body Detection & Segmentation, Body Posture Detection, Depth, Gesture Detection as well as Generic Object Detection – See at least ¶75).
Regarding claim 8, Bigioi, as modified, discloses wherein the memory device is further configured to communicate with the host system in accordance with a serial communication protocol (Serial signaling between the peripheral memory/processing device and the host using conventional serial communication protocols, including SDIO, eMMC, SATA, USB and SPI – See at least ¶30, 77 and 83).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Bigioi et al., US 20190065410A1 in view of Dunne et al., US 20200272899A1, as applied to claim 1 above and further in view of Lee et al., US 20200311546A1, hereinafter referred to as Bigioi, Dunne and Lee, respectively.
Regarding claim 2, the combination of Bigioi and Dunne fail to disclose wherein the evaluation is performed using a third portion of the ANN stored at the host system.
However, Lee teaches wherein the evaluation is performed using a third portion of the ANN stored at the host system (Partitioning a DNN between an edge device and a cloud host. The edge device processes input through a local set of DNN layers and sends the resulting intermediate operation result to the cloud. The cloud uses the received operation result as input to subsequent DNN layers stored and executed at the cloud to obtain a final inference result – See at least ¶38-40. Examiner notes the clouds subsequent DNN layers correspond to the claimed third portion of the ANN, and processing the received result through those layers correspond to evaluating the result through those layers correspond to evaluating the result using that third portion).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Bigioi and Dunne and include the feature of wherein the evaluation is performed using a third portion of the ANN stored at the host system, as taught by Lee, for optimal distributed computing in an edge computing environment.
Claims 3 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Bigioi et al., US 20190065410A1 in view of Dunne et al., US 20200272899A1, as applied to claim 1 above and further in view of Kwon et al., US 20200218979A1, hereinafter referred to as Bigioi, Dunne and Kwon, respectively.
Regarding claim 3, the combination of Bigioi and Dunne fail to disclose wherein the sensor is a sonar sensor.
However, Kwon teaches wherein the sensor is a sonar sensor (using one or more sensors, such as SONAR sensors – See at least ¶8).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Bigioi and Dunne and include the feature of wherein the sensor is a sonar sensor, as taught by Kwon, because sonar provides range and depth information suitable for object detection, particularly when visual sensing is impaired.
Regarding claim 7, the combination of Bigioi and Dunne fail to disclose a control for at least one of steering, braking, or acceleration of a vehicle, wherein the vehicle generates input for the control based on the inference results.
However, Kwon teaches a control for at least one of steering, braking, or acceleration of a vehicle, wherein the vehicle generates input for the control based on the inference results (DNN generated object and distance information used for longitudinal and lateral vehicle control functions. Vehicle controllers configured to send signals to brake actuators, steering actuators, and throttle or accelerator components – See at least ¶2-8 and 181).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Bigioi and Dunne and include the feature of a control for at least one of steering, braking, or acceleration of a vehicle, wherein the vehicle generates input for the control based on the inference results, as taught by Kwon, in order to provide safe autonomous or assisted vehicle operation.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Bigioi et al., US 20190065410A1 in view of Dunne et al., US 20200272899A1, as applied to claim 1 above and further in view of Weng-Jin Wu, US 20190363122A1, hereinafter referred to as Bigioi, Dunne and Wu, respectively.
Regarding claim 4, the combination of Bigioi and Dunne fail to disclose a package that encapsulates the sensor and the memory device.
However, Wu teaches a package that encapsulates the sensor and the memory device (a semiconductor package comprising an image sensor and a stacked second semiconductor die that may include a memory device or memory sensor. Molding compound that encapsulates the stacked second die and the image sensor, including embodiments in which the molding completely encapsulates both components – See at least ¶40-42).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Bigioi and Dunne and include the feature of a package that encapsulates the sensor and the memory device, as taught by Wu, in order to provide a compact integrated sensor package to avoid damage to the sensors.
Claim(s) 9 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Bigioi et al., US 20190065410A1 in view of Giering et al., US 20170371329A1, hereinafter referred to as Bigioi and Giering, respectively.
Regarding claim 9, Bigioi discloses a system comprising:
a first sensor configured to generate image data (The external interface block also provides a direct connection to various image sensors including: a conventional camera (VIS sensor), a NIR sensitive camera, and a thermal imaging camera – See at least ¶78);
a second sensor configured to generate non-image data (For applications, particularly those concerned with the environment external to a vehicle, still further types of image sensors could also be employed for example, a LIDAR image sensor or RADAR image sensor indeed the image sensors could also be supplemented with signals from microwave sensors – See at least ¶78); and
a controller configured to generate inference results (The host controller in conjunction with the device CPU and PCNN CPU loads input image(s) via the peripheral system memory into shared memory – See at least ¶54. Configuring a PCNN according to a neural network program and weights, processing the input images according to the neural network program and storing the resulting output in memory (i.e. generate inference results) – See at least ¶54-57).
Bigioi fails to disclose a controller configured to generate inference results using the image data and non-image data.
However, Giering teaches generate inference results using the image data and non-image data (Providing Lidar data (i.e. non-image data), video data (i.e. image data) supplemental information as input channels to a deep convolutional neural network (DCNN) – See at least ¶38. Applying the DCNN to the Lidar video inputs to generate classification values (i.e. generate inference results using the image data and non-image data) – See at least ¶39).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bigioi and include the feature of generate inference results using the image data and non-image data, as taught by Giering, in order to provide the neural network with complementary visual and depth information and thereby improve the reliability of object and environmental perception.
Regarding claim 11, Bigioi, as modified, discloses wherein the controller is further configured to use the inference results to identify an object (neural network operations including generic object detection, object recognition and body detection. Using the generated inference results to identify an object – See at least ¶75).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Bigioi et al., US 20190065410A1 in view of Giering et al., US 20170371329A1, as applied to claim 9 above and further in view of Tsishkou et al., US 20190086546A1, hereinafter referred to as Bigioi, Giering and Tsishkou, respectively.
Regarding claim 10, the combination of Bigioi and Giering fail to disclose a converter configured to convert the non-image data to an image form prior to using to generate the inference results.
However, Tsishkou teaches a converter configured to convert the non-image data to an image form prior to using to generate the inference results (obtaining non-image three-dimensional point cloud data from a depth sensor and converting the point cloud coordinates, intensity, height, and distance information into respective two-dimensional spaces. The two-dimensional spaces are combined into a multichannel two-dimensional representation, including an RGB compatible representation, and suppled as image form input to a DNN for classification – See at least ¶14-18, 22-27 and 38-40).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Bigioi and Giering and include the feature of a converter configured to convert the non-image data to an image form prior to using to generate the inference results, as taught by Tsishkou, in order to provide the neural network with complementary visual and depth information and thereby improve the reliability of object and environmental perception.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Bigioi et al., US 20190065410A1 in view of Giering et al., US 20170371329A1, as applied to claim 9 above and further in view of Kwon et al., US 20200218979A1, hereinafter referred to as Bigioi, Giering and Kwon, respectively.
Regarding claim 12, the combination of Bigioi and Giering fail to disclose wherein the controller is further configured to use the inference results to determine a distance to an object.
However, Kwon teaches wherein the controller is further configured to use the inference results to determine a distance to an object (applying sensor or image data to a machine learning model to generate object detection and object distance outputs. Computing depth or distance values corresponding to objects depicted in an image and associating the depth values with the detected objects to determine the distance to an object – See at least ¶110-113, 120-112 and 180-183).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Bigioi and Giering and include the feature of wherein the controller is further configured to use the inference results to determine a distance to an object, as taught by Kwon, to provide range and depth information suitable for object detection, particularly when visual sensing is impaired.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Bigioi et al., US 20190065410A1 in view of Giering et al., US 20170371329A1, as applied to claim 9 above and further in view of Dunne et al., US 20200272899A1, hereinafter referred to as Bigioi, Giering and Dunne, respectively.
Regarding claim 13, the combination of Bigioi and Giering fail to disclose wherein the controller is further configured to receive a portion of an artificial neural network based on an evaluation of the inference results.
However, Dunne teaches wherein the controller is further configured to receive a portion of an artificial neural network based on an evaluation of the inference results (Centralized site/device may generate an updated neural network in operation block by using the datasets and inference results received from the new edge device to retrain the neural network (originally trained in operation block). In operation, the centralized site/device may send the updated neural network information (an updated neural network, updated models, updated weights, etc.) to the edge device – See at least ¶96. In operation block, the edge device may use the information received from the centralized site/device (e.g., portions of the updated neural network, updated models, updated weights, etc.) to update the previously-received neural network (received in operation block) and execute the updated neural network – See at least ¶97. Examiner notes, the updated neural network portion sent by the centralized site/device corresponds to the claimed second portion sent based on evaluation of the inference results).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Bigioi and Giering and include the feature of wherein the controller is further configured to receive a portion of an artificial neural network based on an evaluation of the inference results, as taught by Dunne, in order to update a locally executed neural network based on actual inference performance while reducing an amount of neural network data transmitted to a peripheral device.
Claim(s) 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Bigioi et al., US 20190065410A1, in view of Chou et al., US 20120134260A1, hereinafter referred to as Bigioi and Chou, respectively.
Regarding claim 14, Bigioi discloses a system comprising:
a memory device configured to store inference results (Such peripheral storage devices typically comprise a controller portion including a CPU accessing program memory and volatile memory across a parallel system bus – See at least ¶30. The host controller in conjunction with the device CPU and PCNN CPU loads input image(s) via the peripheral system memory into shared memory – See at least ¶54. Configuring a PCNN according to a neural network program and weights, processing the input images according to the neural network program and storing the resulting output in memory (i.e. generate inference results) – See at least ¶54-57); and
at least one controller configured to:
send the inference results to a host system for processing using a first portion of an artificial neural network (ANN) (The host controller in conjunction with the device CPU and PCNN CPU loads input image(s) via the peripheral system memory into shared memory – See at least ¶54. Configuring a PCNN according to a neural network program and weights, processing the input images according to the neural network program and storing the resulting output in memory (i.e. generate inference results) – See at least ¶54-57. Distributing neural network layers (first portion) among multiple PCNN cores and clusters and communicating the output of one CNN – See at least ¶62-63. The input image data is generated by image sensors connected to the neural network processing device – See at least ¶78); and
receive a second portion of the ANN from the host system (The file system acts as a carrier for input data for the CNNs of the cluster as well as results from the CNNs. The file system is also used by the host to transfer the network data (layer configuration, type, weights, instructions) to the PCNN cluster -See at least ¶29. Writes the network configuration into the CNN network file or subdirectory through a file system API write command – See at least ¶35. The host controller in conjunction with the device CPU and PCNN CPU loads neural network configuration and weights from memory (not shown) into shared memory – See at least ¶53. Examiner notes the host supplied layer configuration and associated weights defining one or more neural network layers correspond to the claimed second portion of the ANN).
Bigioi fails to disclose wherein the host system sends the second portion in response to determining data is being dropped due to communication bandwidth limitations.
However, Chou teaches wherein the host system sends the second portion in response to determining data is being dropped due to communication bandwidth limitations (In case of congestion, as the fill-level of queue total bucket exceeds congested threshold, stability controller may assert traffic controls against (e.g., preempt) one or more low-priority LSPs currently using interface. When the fill-level of queue total bucket drops below uncongested threshold, stability controller may revoke policy changes (e.g., return to original drop threshold parameters for critical packets and original bandwidth reservation for the high-priority LSP) on the remaining LSPs associated with interface – See at least ¶40).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bigioi and include the feature of wherein the host system sends the second portion in response to determining data is being dropped due to communication bandwidth limitations, as taught by Chou, to adjust drop parameters for critical packets and/or re-route lower priority label switched paths (LSPs) to avoid network instability caused by, for example, service limitations in other network nodes.
Regarding claim 15, Bigioi, as modified, discloses a camera that provides sensor data for use in generating the inference results (The external interface block also provides a direct connection to various image sensors including: a conventional camera (VIS sensor), a NIR sensitive camera, and a thermal imaging camera – Se at least ¶78).
Regarding claim 16, Bigioi fails to disclose a host interface, wherein the camera is configured to generate the sensor data at a rate that exceeds a communication bandwidth between the host interface and the host system.
However, Chou teaches a host interface, wherein the camera is configured to generate the sensor data at a rate that exceeds a communication bandwidth between the host interface and the host system (In case of congestion, as the fill-level of queue total bucket exceeds congested threshold, stability controller may assert traffic controls against (e.g., preempt) one or more low-priority LSPs currently using interface. When the fill-level of queue total bucket drops below uncongested threshold, stability controller may revoke policy changes (e.g., return to original drop threshold parameters for critical packets and original bandwidth reservation for the high-priority LSP) on the remaining LSPs associated with interface – See at least ¶40).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bigioi and include the feature of a host interface, wherein the camera is configured to generate the sensor data at a rate that exceeds a communication bandwidth between the host interface and the host system, as taught by Chou, to adjust drop parameters for critical packets and/or re-route lower priority label switched paths (LSPs) to avoid network instability caused by, for example, service limitations in other network nodes.
Regarding claim 17, Bigioi, as modified, discloses wherein the host system accesses the inference results by sending commands to the host interface (a host access through file system API commands – See at least ¶28-35).
Regarding claim 18, Bigioi, as modified, discloses wherein the controller is further configured to receive a write command to store the second portion of the ANN in the memory device (a host fwrite () command that transfers and stores neural network configuration – See at least ¶35, 53).
Regarding claim 19, Bigioi, as modified, discloses wherein the controller is further configured to receive a read command requesting access to the inference results (retrieving locally stored output information through a host read command – See at least ¶45-46).
Regarding claim 20, Bigioi, as modified, discloses wherein the host system is further configured to send the second portion in response to evaluating the inference results (host sending an updated ANN portion based on evaluation of received inference results – See at least ¶96-97).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAHMOUD M KAZIMI whose telephone number is (571)272-3436. The examiner can normally be reached M-F 7am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Erin Bishop can be reached at 5712703713. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MAHMOUD M KAZIMI/Examiner, Art Unit 3665
Prosecution Insights