DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 5/11/2026 has been entered.
Response to Amendment
Applicant’s Amendment and remarks submitted on 5/11/2026 have been considered. Claims 24-26 have been added. Claims 1, 5-11, and 21-26 are pending.
Claim Objections. The objections to claims 1 and 21 are withdrawn in view of Applicant’s amendments to claim 1.
35 U.S.C. 112(a) Rejections. The rejections to claim 1 and 5-11 under 35 U.S.C. 112(a) are withdrawn in view of the amendments to claim 1. However, new rejections under 35 U.S.C. 112(a) with respect to claims 1 and 5-11 and 24-25 are set forth herein. The rejections to claim 21-23 under 35 U.S.C. 112(a) are maintained. New claim 26 is also rejected under 35 U.S.C. 112(a) as set forth herein.
Response to Arguments
On pages 9-11 of Applicant’s 5/11/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 112(a), Applicant makes several arguments with respect to claim 1.
The examiner respectfully submits that with respect to claim 1, those arguments are moot because Applicant has deleted the limitations identified by the examiner as lacking written description support. However, new rejections to claim 1 under 35 U.S.C. 112(a) are provided below in view of Applicant’s amended claim language.
On pages 9-11 of Applicant’s 5/11/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 112(a), with respect to claim 21, Applicant relies heavily on para. 0157 of the instant specification, and further on paras. 00112-00116, 00121-00129, 00131-00137, 00153-00159, 00157, and 00212-00214 to show written description support for the “prompt a first processor to analyze the extracted portion of the sensor data to identify a data type of the extracted portion of the sensor data to determine whether to waken a second processor to further analyze the extracted portion of the sensor data and to enable the second processor to select an associated neural network for further processing the extracted portion of the sensor data” limitation.
The examiner respectfully disagrees. While para. 00157 does show that there is an association, or correspondence, between a second neural network, a type of the second target data, and a second external sensor, the specification is silent as to any particular determination of a data type, and then based on such identification, “select an associated neural network for further processing.” Indeed, the instant specification does not appear to provide any logic or criteria for how a particular neural network is to be selected.
Similarly, Applicant cites to paras. 00112-00116, 00121-00129, 00131-00137, 00153-00159, and 00212-00214 as allegedly providing written support. The examiner respectfully disagrees. While these paragraphs may reflect that there is a natural association, or correspondence, between the sensor type and the type of data captured, there is still nothing in the instant specification about determining a particular data type (either by using the sensor type, or other means), and then using such determined data type to select a particular neural network.
On pages 11-20 of Applicant’s 5/11/2026 Amendment and remarks, with respect to the rejection to claim 1 under 35 U.S.C. 103, Applicant makes several arguments with respect to the BETH and DEISHER references.
The examiner respectfully submits that Applicant’s arguments are moot in view of the substantial revisions to claim 1, which necessitated a new ground of rejection in view of the HAN and KOLEN references.
On pages 17-20 of Applicant’s 5/11/2026 Amendment and remarks, with respect to the rejection to claim 1 under 35 U.S.C. 103, Applicant argues that KOLEN is not analogous art.
First, the examiner respectfully submits that Applicant’s specific arguments with respect to the KOLEN reference are largely moot because they pertain to limitations that Applicant has deleted from claim 1.
Second, in response to applicant's general argument that KOLEN is nonanalogous art, it has been held that a prior art reference must either be in the field of the inventor’s endeavor or, if not, then be reasonably pertinent to the particular problem with which the inventor was concerned, in order to be relied upon as a basis for rejection of the claimed invention. See In re Oetiker, 977 F.2d 1443, 24 USPQ2d 1443 (Fed. Cir. 1992). In this case, KOLEN is now utilized to teach the “select a neural network stored in memory based the determined type of data to perform the target operation” limitation with respect to claim 1, and KOLEN clearly teaches this at least at paras. 0062, 0070, ad 0085. To the extent there is a problem that a particular neural network needs to be selected, from more than one possible neural networks, based on a particular input type, KOLEN is reasonably pertinent to that particular problem.
Third, KOLEN relates to neural networks in a software space, where multiple processors are available (see at least, para. 0136). Therefore, KOLEN is also in the same field of endeavor of this application. See MPEP 2141.01(a) (citing Netflix, Inc. v. DivX, LLC, 80 F.4th 1352, 1358-59, 2023 USPQ2d 1057 (Fed. Cir. 2023) ("The field of endeavor is ‘not limited to the specific point of novelty, the narrowest possible conception of the field, or the particular focus within a given field.’") (quoting Unwired Planet, LLC v. Google Inc., 841 F.3d 995, 1001, 120 USPQ2d 1593, 1597 (Fed. Cir. 2016)). When looking at the circumstance of needing criteria for selecting a neural network, one of ordinary skill in the art would reasonably consider other prior art relating to neural networks and selection thereof.
Finally, as argued by Applicant with respect to the 35 U.S.C. 112(a) rejections, if this “select a neural network stored in memory based the determined type of data to perform the target operation” limitation merely requires an association, or correspondence, between a neural network and data type, such correspondence is clearly taught by HAN in Fig. 1, where a particular CNN (130) is used for vision processing, and a particular DNN (134) is used for audio processing.
On page 20 of Applicant’s 5/11/2026 Amendment and remarks, Applicant argues that “Independent claims 21 and 26 have elements similar to those of claim 1 and are believed to be allowable for the same reasons.”
The examiner respectfully disagrees. Independent claims 21 and 26 are substantially different than claim 1, and Applicant’s arguments with respect to claim 1 are not persuasive.
Claim Objections
Claims 1, 11, and 26 are objected to because of the following informalities:
In claim 1, line 14, “wakeup-a non-real-time response processor” should read “wakeup a non-real-time response processor” to remove the extraneous “-“
In claim 1, line 16, “based the determined type” should read “based on the determined type”
In claim 11, lines 1-2, “the first external sensor” lacks antecedent basis. The examiner suggests amending this to recite “the first sensor”
In claim 26, line 5, “selectable neural networks” should read “selectable neural networks;” to add the semicolon delimiter.
Appropriate correction is required.
Claim Rejections - 35 USC § 112(a)
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 1, 5-11, and 21-26 are rejected under 35 U.S.C. 112(a) as failing to comply with the written description requirement. The claims contain subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, at the time the application was filed, had possession of the claimed invention.
In claim 1, the written description does not provide support for the newly-added “select a neural network stored in memory based the determined type of data to perform the target operation” limitation. The disclosure does not appear to have any teachings relating to determining a type of data, and does not appear to have any teachings relating to “selecting a neural network” based on such data type, as explained further in the Response to Arguments section. One of ordinary skill would therefore not understand the inventors to have possession of this limitation as of the effective filing date of the application.
Claims 5-11 and 24-25 depend from claim 1, do not remedy the deficiencies of claim 1, and are therefore rejected for the same reasons explained with respect to claim 1.
In claim 21, the written description does not provide support for the newly-added “prompt the first processor to analyze the extracted portion of the sensor data to identify a data type of the extracted portion of the sensor data to determine whether to waken the second processor to further analyze the extracted portion of the sensor data and to enable the second processor to select an associated neural network stored in memory for further processing the extracted portion of the sensor data based on the data type using the selected neural network” limitation. The disclosure does not appear to have any teachings relating to determining a type of data, and does not appear to have any teachings relating to “select an associated neural network stored in memory for further processing the extracted portion of the sensor data based on the data type using the selected neural network”. One of ordinary skill would therefore not understand the inventors to have possession of this limitation as of the effective filing date of the application.
Claims 22-23 depend from claim 21, do not remedy the deficiencies of claim 21, and are therefore rejected for the same reasons explained with respect to claim 21.
In claim 26, the written description does not provide support for the newly-added “to select one of the plurality of neural networks to perform the target operation” limitation. The disclosure does not appear to have any teachings relating to how a particular neural networks is selected. One of ordinary skill would therefore not understand the inventors to have possession of this limitation as of the effective filing date of the application.
In claim 26, the written description does not provide support for the newly-added “using a corresponding selected neural network based on the first or second data type” limitation. The disclosure does not appear to have any teachings relating to determining a type of data, and does not appear to have any teachings with respect to selecting a particular, corresponding neural network based on data type. One of ordinary skill would therefore not understand the inventors to have possession of this limitation as of the effective filing date of the application.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 26 is rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim 26 recites “when the first identification result is of a second type of data, the real-time response processor and the neural network accelerator working together to produce a first identification result to the non real-time response processor, to enable the non real-time response processor to determine a target operation.” It is not clear if this is an alternative limitation to the “when the first identification result is of a first type of data” limitation, or if this limitation can also be performed in addition to such limitation. If this can be in addition to the other limitation, then the terms “a first identification result” and “a target operation” are duplicative to the same terms identified previously in the claim, so it’s unclear if the “first identification result” and “target operation” are supposed to override, or supplement, the previously determined “first identification result” and “target operation”. Or should these terms relate to “a first identification result for the second type of data” and “a target operation for the second type of data.” Because the metes and bounds of this limitation are not clear, the claim is indefinite and therefore rejected. MPEP 2173.02.
Claim 26 recites “when the real-time response processor coupled to receive the first sensor data and second sensor data, ....” It is unclear if this is meant to be a conditional limitation, such that the remainder of this limitation only needs to be satisfied “when the real-time response processor is coupled to receive the first sensor data and second sensor data”, or if the coupling by the real-time response processor is mandatory in all cases. Because the metes and bounds of this limitation are not clear, the claim is indefinite and therefore rejected. MPEP 2173.02.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 11, and 24-26 are rejected under 35 U.S.C. 103 as being unpatentable over US 20210232199 A1, hereinafter referenced as HAN, in view of US 20200306632 A1, hereinafter referenced as KOLEN.
Regarding Claim 1
HAN teaches:
An integrated circuit for processing sensor data, comprising: (HAN, Fig. 1, context sensor hub 100;
HAN, para. 0017: “FIG. 1 is a block diagram of an example hardware architecture for an example context sensor hub 100 constructed in accordance with teachings disclosed herein. In the illustrated example, the context sensor hub 100 includes five separate processor cores ...”)
an input port coupled to receive sensor first sensor data from a first sensor; (HAN, Fig. 1, I/O interfaces 118 and 120;
HAN, para. 0017: “Further, in some examples, one or more input/output (I/O) interfaces 118 are provided in the ultra-low power domain 112 while one or more other I/O interfaces 120 are provided in the low power domain 114. The I/O interfaces 118, 120 enable the context sensor hub 100 to interface with and receive sensor data from sensors or other hardware peripherals of an associated electronic device. Some example sensors include an accelerometer, a gyroscope, a magnetometer, an ambient light sensor, a proximity sensor (e.g., an IR sensor), a microphone, location sensors (e.g., GNSS sensor, Wi-Fi receiver, etc.), an image sensor (e.g., a camera), etc.”)
a neural network accelerator; (HAN, Fig. 1, CNN Hardware Accelerator 130;
HAN, para. 0020: “. The example VP offload engine 106 includes a VP DSP 128 specifically designed to perform computationally intensive vision analysis such as, for example object (e.g., facial) recognition. The VP offload engine 106 also includes a convolution neural network (CNN) accelerator 130 to facilitate object recognition processing.”)
a non real-time response processor; (HAN, Fig. 1, Vision Processing DSP 128;
HAN, para. 0020: “The example VP offload engine 106 includes a VP DSP 128 specifically designed to perform computationally intensive vision analysis such as, for example object (e.g., facial) recognition. The VP offload engine 106 also includes a convolution neural network (CNN) accelerator 130 to facilitate object recognition processing.”)
a real-time response processor coupled to receive first data, the real-time response processor configured to; (HAN, Fig. 1, main host microcontroller 102;
HAN, para. 0013: “In some examples, at least one core operates in the ultra-low power domain and is always powered (e.g., never power gated) when an associated electronic device is not fully turned off. By contrast, other cores operate in the low power domain and remain asleep or deactivated unless additional computational capacity is needed whereupon they may be woken up. In some examples, these low power cores are awakened by the ultra-low power core in response to an event detected based on sensor data being monitored and/or processed by the ultra-low power core. In some examples, one low power core may wake up a different low power core.”;
HAN, para. 0019: “General management of the context sensor hub 100 is provided by the example host controller 102. Thus, in some examples, when a user (e.g., an OEM) seeks to configure the sensor hub 100, the user interfaces directly with the host controller 102 and then the host controller 102 may pass configuration data onto the other cores. Further, in some examples, most of the drivers for sensors and/or other hardware peripherals monitored by the sensor hub 100 are included on the host controller.”)
extract first target data from the first sensor data wherein the first target data is a portion of the first sensor data, (HAN, para. 0032: “As an example, the host controller 102 may execute vision logic that calls the API proxy 310 to offload a particular image processing task (e.g., image cropping, image scaling, etc.) to the VP DSP 128 (FIG. 1) of the VP offload engine 106. Further, the vision logic of the host controller 102 may call the API proxy 310 a second time to open and run a CNN model on the cropped image for object recognition.”;
Examiner’s Note: image data corresponds to recited “first sensor data” and the cropped image corresponds to the recited “first target data”)
send the first target data to the neural network accelerator, and (HAN, para. 0032: “Further, the vision logic of the host controller 102 may call the API proxy 310 a second time to open and run a CNN model on the cropped image for object recognition. In some examples, the API proxy 310 passes this request to the VP offload engine 106 to perform the object recognition using the VP DSP 128 and the CNN accelerator 130.”)
when the real-time processor receives a first identification result sent by the neural network accelerator in real time: (HAN, para. 0020: “The VP offload engine 106 also includes a convolution neural network (CNN) accelerator 130 to facilitate object recognition processing.”
HAN, para. 0032: “Further, the vision logic of the host controller 102 may call the API proxy 310 a second time to open and run a CNN model on the cropped image for object recognition. In some examples, the API proxy 310 passes this request to the VP offload engine 106 to perform the object recognition using the VP DSP 128 and the CNN accelerator 130.”;
Examiner’s Note: host controller 102 receives an object recognition (corresponding to recited “first identification result” from the CNN accelerator 130)
determine a corresponding type of data, (HAN, para. 0020: “The ISP 126 facilitates the capturing of such images that may then be processed by the VCD DSP 122. The example VP offload engine 106 includes a VP DSP 128 specifically designed to perform computationally intensive vision analysis such as, for example object (e.g., facial) recognition. The VP offload engine 106 also includes a convolution neural network (CNN) accelerator 130 to facilitate object recognition processing.”)
determine a target operation based on the first identification result; (HAN, para. 0020: “The VP offload engine 106 also includes a convolution neural network (CNN) accelerator 130 to facilitate object recognition processing.”
HAN, para. 0032: “Further, the vision logic of the host controller 102 may call the API proxy 310 a second time to open and run a CNN model on the cropped image for object recognition. In some examples, the API proxy 310 passes this request to the VP offload engine 106 to perform the object recognition using the VP DSP 128 and the CNN accelerator 130.”;
Examiner’s Note: facial objection recognition corresponds to recited “target operation”)
determine to wakeup-a non real-time response processor based on the determined data type to perform the target operation; and (HAN, para. 0013: “Furthermore, examples disclosed herein enable different ones of the cores to be put to sleep when not currently being used and activated or awakened when the particular functionality offered by the cores is needed. Thus, while the cores are considered always-on as the term is defined herein (e.g., can be active and operating when the associated electronic device is in a sleep or idle mode), not all of the cores are necessarily always powered. ... In some examples, one low power core may wake up a different low power core.”;
HAN, para. 0014: “Among other things, distributing functionality across multiple cores makes it difficult to control the separate cores and/or enable their efficient interaction (e.g., to wake a particular core up when needed and/or to put a particular core to sleep when no longer need).”;
HAN, para. 0020: “In the illustrated example, the host controller 102 offloads more computationally intensive tasks to one or more of the three other cores 104, 106, 108 (collectively referred to herein as offload engines). As shown in FIG. 1, the VCD offload engine 104 includes a VCD digital signal processor (DSP) 122, an I/O interface 124, and a low power image signal processor (ISP) 126. The I/O interface 124 is to communicate (e.g., interface) with a camera for capturing images. The ISP 126 facilitates the capturing of such images that may then be processed by the VCD DSP 122. The example VP offload engine 106 includes a VP DSP 128 specifically designed to perform computationally intensive vision analysis such as, for example object (e.g., facial) recognition. The VP offload engine 106 also includes a convolution neural network (CNN) accelerator 130 to facilitate object recognition processing.”;
Examiner’s Note: VP DSP 128 is computationally intensive and is awoken by the host microcontroller 102 when needed)
select a neural network stored in memory based the determined type of data to perform the target operation; and
perform, by the non real-time response processor, the target operation using the selected neural network.
However, HAN fails to explicitly teach:
select a neural network stored in memory based the determined type of data to perform the target operation; and
perform, by the non real-time response processor, the target operation using the selected neural network.
However, in a related field of endeavor (adjusting computing resources, see para. 0042), KOLEN teaches:
select a neural network stored in memory based the determined type of data to perform the target operation; and (KOLEN, para. 0062: “Some non-limiting examples of machine learning algorithms that can be used to generate and update the prediction models can include supervised and non-supervised machine learning algorithms, including ... artificial neural network algorithms (such as, for example, Perceptron), deep learning algorithms (such as, for example, Deep Boltzmann Machine), ... and/or other machine learning algorithms.”;
KOLEN, para. 0070: “The user computing system 110 may have varied local computing resources, such as central processing units and architectures, memory, mass storage, graphics processing units, communication network availability and bandwidth, and so forth. Further, the user computing system 110 may include any type of computing system.”
KOLEN, para. 0085: “The computing resource adjustment system 118 can determine which prediction model, such as any of models 160A-N, to use based on input data 172 and/or additional identifiers associated with the input data 172. Additionally, the selection of a prediction model 160 may be based on the specific input data 172 provided. The availability of particular types of data as part of the input data 172 can affect the selection of the prediction model 160.”;
Examiner’s Note: KOLEN discloses selecting a specific prediction model (which can be based on neural network algorithms) based on a specific type of input data; the HAN-KOLEN combination now modifies HAN so that a specific type of neural network can be selected based on the input type, e.g., different neural networks for digital camera imagery vs. audio inputs)
perform, by the non real-time response processor, the target operation using the selected neural network. (KOLEN, para. 0062: “Some non-limiting examples of machine learning algorithms that can be used to generate and update the prediction models can include supervised and non-supervised machine learning algorithms, including ... artificial neural network algorithms (such as, for example, Perceptron), deep learning algorithms (such as, for example, Deep Boltzmann Machine), ... and/or other machine learning algorithms.”;
KOLEN, para. 0085: “The computing resource adjustment system 118 can determine which prediction model, such as any of models 160A-N, to use based on input data 172 and/or additional identifiers associated with the input data 172. Additionally, the selection of a prediction model 160 may be based on the specific input data 172 provided. The availability of particular types of data as part of the input data 172 can affect the selection of the prediction model 160.”;
Examiner’s Note: the HAN-KOLEN combination now modifies HAN so that the Vision Processing DSP 128 of HAN, using the CNN Hardware Accelerator 130, does the facial recognition of HAN using the selected neural network based on KOLEN)
Before the effective filing date of the present application, it would have been obvious to a person of ordinary skill in the art to combine the teachings of HAN with KOLEN as explained herein. As disclosed by KOLEN, one of ordinary skill in the art would have been motivated to do so in order to predict computing resources so that “unnecessary delays associated with acquiring additional resources can be avoided or reduced.” (para. 0026).
Regarding Claim 11
HAN and KOLEN disclose the integrated circuit according to claim 1. HAN further teaches:
wherein the first external sensor comprises at least one of a camera, a microphone, a motion sensor, a distance sensor, an ambient optical sensor, a magnetic field sensor, a fingerprint sensor, or a temperature sensor. (HAN, para. 0017: “Some example sensors include an accelerometer, a gyroscope, a magnetometer, an ambient light sensor, a proximity sensor (e.g., an IR sensor), a microphone, location sensors (e.g., GNSS sensor, Wi-Fi receiver, etc.), an image sensor (e.g., a camera), etc.”;
HAN, para. 0025: “ Example sensors that may be implemented as always running or at least as always-on (but not necessarily always running) include an accelerometer, a gyroscope, a magnetometer, an ambient light sensor, a proximity sensor (e.g., an IR sensor), and a microphone.”)
Regarding Claim 24
HAN and KOLEN disclose the integrated circuit according to claim 1. HAN further teaches:
wherein the real-time response processor is further configured to operate according to a third sequence, wherein the third sequence performed by the real-time response processor comprises: (HAN, para. 0019: “General management of the context sensor hub 100 is provided by the example host controller 102. Thus, in some examples, when a user (e.g., an OEM) seeks to configure the sensor hub 100, the user interfaces directly with the host controller 102 and then the host controller 102 may pass configuration data onto the other cores. Further, in some examples, most of the drivers for sensors and/or other hardware peripherals monitored by the sensor hub 100 are included on the host controller.”)
extract the first target data from the first sensor data wherein the first target data is a portion of the first sensor data, (HAN, para. 0032: “As an example, the host controller 102 may execute vision logic that calls the API proxy 310 to offload a particular image processing task (e.g., image cropping, image scaling, etc.) to the VP DSP 128 (FIG. 1) of the VP offload engine 106. Further, the vision logic of the host controller 102 may call the API proxy 310 a second time to open and run a CNN model on the cropped image for object recognition.”;
Examiner’s Note: image data corresponds to recited “first sensor data” and the cropped image corresponds to the recited “first target data”)
send the first target data to the neural network accelerator, (HAN, para. 0032: “Further, the vision logic of the host controller 102 may call the API proxy 310 a second time to open and run a CNN model on the cropped image for object recognition. In some examples, the API proxy 310 passes this request to the VP offload engine 106 to perform the object recognition using the VP DSP 128 and the CNN accelerator 130.”)
indicate to the non real-time processor to wake up by sending a second identification result to the non real-time processor. (HAN, para. 0013: “Furthermore, examples disclosed herein enable different ones of the cores to be put to sleep when not currently being used and activated or awakened when the particular functionality offered by the cores is needed. Thus, while the cores are considered always-on as the term is defined herein (e.g., can be active and operating when the associated electronic device is in a sleep or idle mode), not all of the cores are necessarily always powered. ... In some examples, one low power core may wake up a different low power core.”;
HAN, para. 0014: “Among other things, distributing functionality across multiple cores makes it difficult to control the separate cores and/or enable their efficient interaction (e.g., to wake a particular core up when needed and/or to put a particular core to sleep when no longer need).”;
HAN, para. 0020: “In the illustrated example, the host controller 102 offloads more computationally intensive tasks to one or more of the three other cores 104, 106, 108 (collectively referred to herein as offload engines). As shown in FIG. 1, the VCD offload engine 104 includes a VCD digital signal processor (DSP) 122, an I/O interface 124, and a low power image signal processor (ISP) 126. The I/O interface 124 is to communicate (e.g., interface) with a camera for capturing images. The ISP 126 facilitates the capturing of such images that may then be processed by the VCD DSP 122. The example VP offload engine 106 includes a VP DSP 128 specifically designed to perform computationally intensive vision analysis such as, for example object (e.g., facial) recognition. The VP offload engine 106 also includes a convolution neural network (CNN) accelerator 130 to facilitate object recognition processing.”;
Examiner’s Note: VP DSP 128 is computationally intensive and is awoken by the host microcontroller 102 when needed, which can be awoken based on the result as explained by at least paras. 0092-0093 of the instant specification)
Regarding Claim 25
HAN and KOLEN disclose the integrated circuit according to claim 1. HAN further teaches:
wherein the real-time response processor is further configured to operate according to a fourth sequence, wherein the fourth sequence performed by the real-time response processor comprises: (HAN, para. 0019: “General management of the context sensor hub 100 is provided by the example host controller 102. Thus, in some examples, when a user (e.g., an OEM) seeks to configure the sensor hub 100, the user interfaces directly with the host controller 102 and then the host controller 102 may pass configuration data onto the other cores. Further, in some examples, most of the drivers for sensors and/or other hardware peripherals monitored by the sensor hub 100 are included on the host controller.”)
extract the first target data from the first sensor data and second target data from a second sensor wherein the first target data is a portion of the first sensor data and the second target data is a portion of second sensor data, (HAN, para. 0017: “Further, in some examples, one or more input/output (I/O) interfaces 118 are provided in the ultra-low power domain 112 while one or more other I/O interfaces 120 are provided in the low power domain 114. The I/O interfaces 118, 120 enable the context sensor hub 100 to interface with and receive sensor data from sensors or other hardware peripherals of an associated electronic device. Some example sensors include an accelerometer, a gyroscope, a magnetometer, an ambient light sensor, a proximity sensor (e.g., an IR sensor), a microphone, location sensors (e.g., GNSS sensor, Wi-Fi receiver, etc.), an image sensor (e.g., a camera), etc.”;
HAN, para. 0032: “As an example, the host controller 102 may execute vision logic that calls the API proxy 310 to offload a particular image processing task (e.g., image cropping, image scaling, etc.) to the VP DSP 128 (FIG. 1) of the VP offload engine 106. Further, the vision logic of the host controller 102 may call the API proxy 310 a second time to open and run a CNN model on the cropped image for object recognition.”;
Examiner’s Note: image data corresponds to recited “first sensor data” and the cropped image corresponds to the recited “first target data”, and second target data can be proximity sensor data identifying a person in proximity)
send the first and second target data to the neural network accelerator so that the neural network accelerator can produce first and second identification results to the non real-time response processor, and (HAN, para. 0032: “Further, the vision logic of the host controller 102 may call the API proxy 310 a second time to open and run a CNN model on the cropped image for object recognition. In some examples, the API proxy 310 passes this request to the VP offload engine 106 to perform the object recognition using the VP DSP 128 and the CNN accelerator 130.”;
Examiner’s Note: A IR sensor produces infrared images that can be analyzed by the CNN accelerator)
indicate to the non real-time processor to wake up so that it can be ready to process the first and second identification results received from the neural network accelerator. (HAN, para. 0013: “Furthermore, examples disclosed herein enable different ones of the cores to be put to sleep when not currently being used and activated or awakened when the particular functionality offered by the cores is needed. Thus, while the cores are considered always-on as the term is defined herein (e.g., can be active and operating when the associated electronic device is in a sleep or idle mode), not all of the cores are necessarily always powered. ... In some examples, one low power core may wake up a different low power core.”;
HAN, para. 0014: “Among other things, distributing functionality across multiple cores makes it difficult to control the separate cores and/or enable their efficient interaction (e.g., to wake a particular core up when needed and/or to put a particular core to sleep when no longer need).”;
HAN, para. 0020: “In the illustrated example, the host controller 102 offloads more computationally intensive tasks to one or more of the three other cores 104, 106, 108 (collectively referred to herein as offload engines). As shown in FIG. 1, the VCD offload engine 104 includes a VCD digital signal processor (DSP) 122, an I/O interface 124, and a low power image signal processor (ISP) 126. The I/O interface 124 is to communicate (e.g., interface) with a camera for capturing images. The ISP 126 facilitates the capturing of such images that may then be processed by the VCD DSP 122. The example VP offload engine 106 includes a VP DSP 128 specifically designed to perform computationally intensive vision analysis such as, for example object (e.g., facial) recognition. The VP offload engine 106 also includes a convolution neural network (CNN) accelerator 130 to facilitate object recognition processing.”;
Examiner’s Note: VP DSP 128 is computationally intensive and is awoken by the host microcontroller 102 when needed, which can be awoken based on the result as explained by at least paras. 0092-0093 of the instant specification)
Regarding Claim 26
HAN teaches:
An integrated circuit for processing sensor data, comprising: (HAN, Fig. 1, context sensor hub 100;
HAN, para. 0017: “FIG. 1 is a block diagram of an example hardware architecture for an example context sensor hub 100 constructed in accordance with teachings disclosed herein. In the illustrated example, the context sensor hub 100 includes five separate processor cores ...”)
a neural network accelerator; (HAN, Fig. 1, CNN Hardware Accelerator 130;
HAN, para. 0020: “. The example VP offload engine 106 includes a VP DSP 128 specifically designed to perform computationally intensive vision analysis such as, for example object (e.g., facial) recognition. The VP offload engine 106 also includes a convolution neural network (CNN) accelerator 130 to facilitate object recognition processing.”)
a non real-time response processor; (HAN, Fig. 1, Vision Processing DSP 128;
HAN, para. 0020: “The example VP offload engine 106 includes a VP DSP 128 specifically designed to perform computationally intensive vision analysis such as, for example object (e.g., facial) recognition. The VP offload engine 106 also includes a convolution neural network (CNN) accelerator 130 to facilitate object recognition processing.”)
a memory coupled to the non real-time response processor ... (HAN, Fig. 1, shared memory 116;
HAN, para. 0010: “Some such image processing chips implement a convolution neural network (CNN) to enable object detection (e.g. detect the presence of and/or location of an object (e.g., a face) within an image) and/or object recognition (e.g., identify and/or distinguish a detected object from other objects (e.g., recognize a particular person based on their face)).”;
HAN, para. 0017: “The example context sensor hub 100 also includes a shared memory 116 that operates within the low power domain 114.”)
a real-time response processor coupled to receive the first sensor data wherein the integrated circuit processes the first sensor data by: (HAN, Fig. 1, main host microcontroller 102;
HAN, para. 0013: “In some examples, at least one core operates in the ultra-low power domain and is always powered (e.g., never power gated) when an associated electronic device is not fully turned off. By contrast, other cores operate in the low power domain and remain asleep or deactivated unless additional computational capacity is needed whereupon they may be woken up. In some examples, these low power cores are awakened by the ultra-low power core in response to an event detected based on sensor data being monitored and/or processed by the ultra-low power core. In some examples, one low power core may wake up a different low power core.”;
HAN, para. 0019: “General management of the context sensor hub 100 is provided by the example host controller 102. Thus, in some examples, when a user (e.g., an OEM) seeks to configure the sensor hub 100, the user interfaces directly with the host controller 102 and then the host controller 102 may pass configuration data onto the other cores. Further, in some examples, most of the drivers for sensors and/or other hardware peripherals monitored by the sensor hub 100 are included on the host controller.”)
when the first identification result is of a first type of data, the real-time response processor and the neural network accelerator working together to produce a first identification result and based upon the first identification result, determine a target operation and perform the target operation; (HAN, para. 0020: “The ISP 126 facilitates the capturing of such images that may then be processed by the VCD DSP 122. The example VP offload engine 106 includes a VP DSP 128 specifically designed to perform computationally intensive vision analysis such as, for example object (e.g., facial) recognition. The VP offload engine 106 also includes a convolution neural network (CNN) accelerator 130 to facilitate object recognition processing.”;
HAN, para. 0032: “Further, the vision logic of the host controller 102 may call the API proxy 310 a second time to open and run a CNN model on the cropped image for object recognition. In some examples, the API proxy 310 passes this request to the VP offload engine 106 to perform the object recognition using the VP DSP 128 and the CNN accelerator 130.”;
Examiner’s Note: object information recognizes objects from a first type of data, and facial objection recognition corresponds to recited “target operation” that is performed)
when the first identification result is of a second type of data, the real-time response processor and the neural network accelerator working together to produce a first identification result to the non real-time response processor, to enable the non real-time response processor to determine a target operation ...; and (HAN, para. 0017: “Some example sensors include an accelerometer, a gyroscope, a magnetometer, an ambient light sensor, a proximity sensor (e.g., an IR sensor), a microphone, location sensors (e.g., GNSS sensor, Wi-Fi receiver, etc.), an image sensor (e.g., a camera), etc.”;
HAN, para. 0021: “Finally, as shown in FIG. 1, the audio offload engine 108 includes an audio processing DSP 132 specifically designed to perform computationally intensive audio processing and analysis such as, for example, speech detection and/or voice recognition.”
HAN, para. 0025: “ Example sensors that may be implemented as always running or at least as always-on (but not necessarily always running) include an accelerometer, a gyroscope, a magnetometer, an ambient light sensor, a proximity sensor (e.g., an IR sensor), and a microphone.”;
Examiner’s Note: the second type of data can be a microphone sensor to perform voice recognition (corresponding to recited “target operation”)
when the real-time response processor coupled to receive the first sensor data and second sensor data, the real-time response processor produces first and second target data to the neural network accelerator which produces the first and a second identification result to the non real-time response processor, to enable the non real-time response processor to determine a target operation and perform the target operation ... (HAN, para. 0020: “The ISP 126 facilitates the capturing of such images that may then be processed by the VCD DSP 122. The example VP offload engine 106 includes a VP DSP 128 specifically designed to perform computationally intensive vision analysis such as, for example object (e.g., facial) recognition. The VP offload engine 106 also includes a convolution neural network (CNN) accelerator 130 to facilitate object recognition processing.”;
HAN, para. 0032: “Further, the vision logic of the host controller 102 may call the API proxy 310 a second time to open and run a CNN model on the cropped image for object recognition. In some examples, the API proxy 310 passes this request to the VP offload engine 106 to perform the object recognition using the VP DSP 128 and the CNN accelerator 130.”;
Examiner’s Note: the recited “second sensor data” can be a second piece of data from the same sensor, and therefore multiple images from a camera can be used to perform the target facial recognition task)
However, HAN fails to explicitly teach:
... that stores a plurality of selectable neural networks
and to select one of the plurality of neural networks to perform the target operation
using a corresponding selected neural network based on the first or second data type
However, in a related field of endeavor (adjusting computing resources, see para. 0042), KOLEN teaches:
However, HAN fails to explicitly teach:
... that stores a plurality of selectable neural networks (KOLEN, para. 0062: “Some non-limiting examples of machine learning algorithms that can be used to generate and update the prediction models can include supervised and non-supervised machine learning algorithms, including ... artificial neural network algorithms (such as, for example, Perceptron), deep learning algorithms (such as, for example, Deep Boltzmann Machine), ... and/or other machine learning algorithms.”;
KOLEN, para. 0070: “The user computing system 110 may have varied local computing resources, such as central processing units and architectures, memory, mass storage, graphics processing units, communication network availability and bandwidth, and so forth. Further, the user computing system 110 may include any type of computing system.”
KOLEN, para. 0085: “The computing resource adjustment system 118 can determine which prediction model, such as any of models 160A-N, to use based on input data 172 and/or additional identifiers associated with the input data 172. Additionally, the selection of a prediction model 160 may be based on the specific input data 172 provided. The availability of particular types of data as part of the input data 172 can affect the selection of the prediction model 160.”;
Examiner’s Note: KOLEN discloses selecting a specific prediction model (which can be based on neural network algorithms) based on a specific type of input data, and further discloses a memory for storing system resources; the HAN-KOLEN combination now modifies HAN so that a specific type of neural network can be selected based on the input type, e.g., different neural networks for digital camera imagery vs. audio inputs as in KOLEN, where such selectable neural networks are stored in the memory of KOLEN)
and to select one of the plurality of neural networks to perform the target operation (KOLEN, para. 0062: “Some non-limiting examples of machine learning algorithms that can be used to generate and update the prediction models can include supervised and non-supervised machine learning algorithms, including ... artificial neural network algorithms (such as, for example, Perceptron), deep learning algorithms (such as, for example, Deep Boltzmann Machine), ... and/or other machine learning algorithms.”;
KOLEN, para. 0085: “The computing resource adjustment system 118 can determine which prediction model, such as any of models 160A-N, to use based on input data 172 and/or additional identifiers associated with the input data 172. Additionally, the selection of a prediction model 160 may be based on the specific input data 172 provided. The availability of particular types of data as part of the input data 172 can affect the selection of the prediction model 160.”;
Examiner’s Note: KOLEN discloses selecting a specific prediction model (which can be based on neural network algorithms) based on a specific type of input data; the HAN-KOLEN combination now modifies HAN so that a specific type of neural network can be selected based on the input type, e.g., different neural networks for digital camera imagery vs. audio inputs as in KOLEN)
using a corresponding selected neural network based on the first or second data type (KOLEN, para. 0062: “Some non-limiting examples of machine learning algorithms that can be used to generate and update the prediction models can include supervised and non-supervised machine learning algorithms, including ... artificial neural network algorithms (such as, for example, Perceptron), deep learning algorithms (such as, for example, Deep Boltzmann Machine), ... and/or other machine learning algorithms.”;
KOLEN, para. 0085: “The computing resource adjustment system 118 can determine which prediction model, such as any of models 160A-N, to use based on input data 172 and/or additional identifiers associated with the input data 172. Additionally, the selection of a prediction model 160 may be based on the specific input data 172 provided. The availability of particular types of data as part of the input data 172 can affect the selection of the prediction model 160.”;
Examiner’s Note: KOLEN discloses selecting a specific prediction model (which can be based on neural network algorithms) based on a specific type of input data; the HAN-KOLEN combination now modifies HAN so that a specific type of neural network can be selected and used based on the input type, e.g., different neural networks for digital camera imagery vs. audio inputs as in KOLEN)
Before the effective filing date of the present application, it would have been obvious to a person of ordinary skill in the art to combine the teachings of HAN with KOLEN as explained herein. As disclosed by KOLEN, one of ordinary skill in the art would have been motivated to do so in order to predict computing resources so that “unnecessary delays associated with acquiring additional resources can be avoided or reduced.” (para. 0026).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over HAN in view of KOLEN and further in view of US 20190235488 A1, hereinafter referenced as BETH.
Regarding Claim 5
HAN and KOLEN disclose the integrated circuit according to claim 1. HAN further teaches:
wherein the real-time response processor is further configured to: (HAN, Fig. 1, main host microcontroller 102;
HAN, para. 0019: “General management of the context sensor hub 100 is provided by the example host controller 102. Thus, in some examples, when a user (e.g., an OEM) seeks to configure the sensor hub 100, the user interfaces directly with the host controller 102 and then the host controller 102 may pass configuration data onto the other cores. Further, in some examples, most of the drivers for sensors and/or other hardware peripherals monitored by the sensor hub 100 are included on the host controller.”)
obtain second sensor data from a second external sensor, and extract second target data from the second sensor data; and (HAN, para. 0017: “Some example sensors include an accelerometer, a gyroscope, a magnetometer, an ambient light sensor, a proximity sensor (e.g., an IR sensor), a microphone, location sensors (e.g., GNSS sensor, Wi-Fi receiver, etc.), an image sensor (e.g., a camera), etc.”;
HAN, para. 0025: “ Example sensors that may be implemented as always running or at least as always-on (but not necessarily always running) include an accelerometer, a gyroscope, a magnetometer, an ambient light sensor, a proximity sensor (e.g., an IR sensor), and a microphone.”;
Examiner’s Note: the second sensor data can be a microphone sensor to detect the sound of a human nearby)
However, HAN and KOLEN fail to explicitly teach:
the neural network accelerator is further configured to identify the second target data based on a second neural network model to obtain a second identification result, wherein the second identification result and the first identification result are used to determine the target operation together.
However, in a related field of endeavor (a vehicle controller using an AI accelerator and multiple processors, see para. 0021), BETH teaches and makes obvious:
the neural network accelerator is further configured to identify the second target data based on a second neural network model to obtain a second identification result, wherein the second identification result and the first identification result are used to determine the target operation together. (BETH, para. 0025: “In embodiments, the vehicles 210A, 210B may include an incident detailed examination neural network (IDENN) 230, which may be used to detect relevant events and identify areas of interest relevant to a mission plan of the vehicle. The IDENN 230 may be enabled to quickly (e.g., in near real-time) and efficiently (e.g., using fewer processing cycles than existing technology) process the data generated from the vehicle sensors (e.g., digital cameras 222, LIDAR 218, IR sensors 220, and the like) and optionally from external sensors and/or data sources to detect issues during the vehicle's mission path. The IDENN 230 can trigger a modification of a planned mission path to provide a closer and/or more in-depth look at an identified issue. The IDENN 230 can then use the new data from the closer look to verify the issue, acquire more data if necessary, and create a data-capture report.”
BETH, para. 0026: “More specifically, in embodiments, upon determination of an incident or event, a data capture neural network (DCNN) 232 may be activated. The DCNN 232 may be used to provide a continuously improving data-capture that maximizes efficiency of the planned path geometry considering both the amount of data needed, the environmental conditions, and the optimal viewing angles of the sensors.”;
BETH, para. 0030: “Edge processing allows a vehicle 210A, 210B to gather critical data relating to a mission by fusing the feeds of multiple disparate sensors while using neural networks to process a data feed in real-time for incident/event detection or classification. Incorporated on top of the vehicle's onboard real-time controller 236 is the main CPU integrated sensor feed data. Processing acquired data in real-time and determining critical (red-flagged) events may trigger further investigation and data acquisition.”;
Examiner’s Note: BETH discloses fusing sensor feeds so that a neural network can make real-time decisions for incident/event detection or classification; the HAN-KOLEN-BETH combination now modifies HAN so that multiple sensor feeds can be used together to make classifications as in BETH, such as using both image data and infrared data for facial recognition)
Before the effective filing date of the present application, it would have been obvious to a person of ordinary skill in the art to combine the teachings of HAN with KOLEN and BETH as explained herein. As disclosed by BETH, one of ordinary skill in the art would have been motivated to do so in order to permit a “reduced dataset of relevant information to be provided to the cloud server and ultimately to an end-user.” (para. 0030). In other words, the ability to fuse multiple disparate sensor sensors locally, without needing to send such information to a cloud server, has advantages at least of lower data transmission costs.
Claims 6-8 are rejected under 35 U.S.C. 103 as being unpatentable over HAN in view of KOLEN and BETH and further in view of Yu, Qi, et al. "A deep learning prediction process accelerator based FPGA." 2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing. IEEE, 2015, pp. 1159-1162, hereinafter referenced as YU.
Regarding Claim 6
HAN, KOLEN, and BETH disclose the integrated circuit according to claim 5. However, HAN, KOLEN, and BETH fail to explicitly teach:
wherein the neural network accelerator is further configured to identify the first target data and the second target data in a time-sharing manner.
However, in a related field of endeavor (deep learning and neural networks for identification and recognition, see p. 1159, section I), YU teaches:
wherein the accelerator is further configured to identify the first target data and the second target data in a time-sharing manner. (YU, p. 1159, section I: “In this paper, we also use FPGA to design the hardware accelerator for deep learning prediction process. For large scale neural networks where direct mapping is not possible, the implementation becomes a problem in terms of performance and the hardware resources. To tackle this problem, we use time-sharing reused technology.”; YU, p. 1161, section III.B: “we decompose the input data into data fragments and use time-sharing multiplexing technology to deal with these fragments”; (EN): in combination with HAN, KOLEN, and BETH, the system of HAN now updates its CNN accelerator using the accelerator techniques of YU which utilizes time-sharing multiplexing)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of HAN, KOLEN, BETH, and YU as explained above. As disclosed by YU, one of ordinary skill would have been motivated to do so the accelerator of YU was experimentally shown to “achieve average 30x speedup.” (p. 1152, section VI).
Regarding Claim 7
HAN, KOLEN, BETH, and YU disclose the integrated circuit according to claim 6. However, HAN and KOLEN fail to explicitly teach:
a controller, configured to determine a first priority corresponding to the first target data and a second priority corresponding to the second target data; and
the neural network accelerator is configured to identify, based on the first priority and the second priority, the first target data and the second target data in a time-sharing manner.
However, in a related field of endeavor (a vehicle controller using an AI accelerator and multiple processors, see para. 0021), BETH teaches and makes obvious:
a controller, configured to determine a first priority corresponding to the first target data and a second priority corresponding to the second target data; and (BETH, Fig. 2, Controller 236, and para. 0021: “The vehicle controller 236 may include a processing element such as a GPU, floating point processor, AI accelerator and the like for performing vehicle and related data processing”; BETH, para. 0035: “The vehicle may classify data in order to prioritize the data to transmit.”; (EN): Controller 236 includes a filtering module 242, which prioritizes data for transmission, e.g., first target data might have a higher priority for transmission than the second target data)
Before the effective filing date of the present application, it would have been obvious to a person of ordinary skill in the art to combine the teachings of HAN with KOLEN, BETH, and YU as explained herein. As disclosed by BETH, one of ordinary skill in the art would have been motivated to do so in order to permit a “reduced dataset of relevant information to be provided to the cloud server and ultimately to an end-user.” (para. 0030). In other words, the ability to fuse multiple disparate sensor sensors locally, without needing to send such information to a cloud server, has advantages at least of lower data transmission costs.
However, HAN, KOLEN, and BETH fail to explicitly teach:
the neural network accelerator is configured to identify, based on the first priority and the second priority, the first target data and the second target data in a time-sharing manner.
However, in a related field of endeavor (deep learning and neural networks for identification and recognition, see p. 1159, section I), YU teaches:
the neural network accelerator is configured to identify, based on the first priority and the second priority, the first target data and the second target data in a time-sharing manner. (YU, p. 1159, section I: “In this paper, we also use FPGA to design the hardware accelerator for deep learning prediction process. For large scale neural networks where direct mapping is not possible, the implementation becomes a problem in terms of performance and the hardware resources. To tackle this problem, we use time-sharing reused technology.”; YU, p. 1161, section III.B: “we decompose the input data into data fragments and use time-sharing multiplexing technology to deal with these fragments”; (EN): in combination with HAN, KOLEN, and BETH, HAN now updates its CNN accelerator to use the accelerator techniques of YU which utilizes time-sharing multiplexing, so that based on the first and second priority (as explained above), first and second target data is identified for transmission based on such priorities as in BETH)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of HAN, KOLEN, BETH, and YU as explained above. As disclosed by YU, one of ordinary skill would have been motivated to do so the accelerator of YU was experimentally shown to “achieve average 30x speedup.” (p. 1152, section VI).
Regarding Claim 8
HAN, KOLEN, BETH, and YU disclose the integrated circuit according to claim 6. However, HAN and KOLEN fail to explicitly teach:
a controller, configured to: determine a first priority corresponding to the first target data and a second priority corresponding to the second target data, and
control, based on the first priority and the second priority, the real-time response processor to send the first target data and the second target data to the accelerator in a time-sharing manner, so that the accelerator identifies the first target data and the second target data in a time-sharing manner.
However, in a related field of endeavor (a vehicle controller using an AI accelerator and multiple processors, see para. 0021), BETH teaches and makes obvious:
a controller, configured to: determine a first priority corresponding to the first target data and a second priority corresponding to the second target data, and (BETH, Fig. 2, Controller 236, and para. 0021: “The vehicle controller 236 may include a processing element such as a GPU, floating point processor, AI accelerator and the like for performing vehicle and related data processing”; BETH, para. 0035: “The vehicle may classify data in order to prioritize the data to transmit.”; (EN): Controller 236 includes a filtering module 242, which prioritizes data for transmission, e.g., first target data might have a higher priority for transmission than the second target data)
Before the effective filing date of the present application, it would have been obvious to a person of ordinary skill in the art to combine the teachings of HAN with KOLEN, BETH, and YU as explained herein. As disclosed by BETH, one of ordinary skill in the art would have been motivated to do so in order to permit a “reduced dataset of relevant information to be provided to the cloud server and ultimately to an end-user.” (para. 0030). In other words, the ability to fuse multiple disparate sensor sensors locally, without needing to send such information to a cloud server, has advantages at least of lower data transmission costs.
However, HAN, KOLEN, and BETH fail to explicitly teach:
control, based on the first priority and the second priority, the real-time response processor to send the first target data and the second target data to the accelerator in a time-sharing manner, so that the accelerator identifies the first target data and the second target data in a time-sharing manner.
However, in a related field of endeavor (deep learning and neural networks for identification and recognition, see p. 1159, section I), YU teaches:
control, based on the first priority and the second priority, the real-time response processor to send the first target data and the second target data to the neural network accelerator in a time-sharing manner, so that the neural network accelerator identifies the first target data and the second target data in a time-sharing manner. (YU, p. 1159, section I: “In this paper, we also use FPGA to design the hardware accelerator for deep learning prediction process. For large scale neural networks where direct mapping is not possible, the implementation becomes a problem in terms of performance and the hardware resources. To tackle this problem, we use time-sharing reused technology.”; YU, p. 1161, section III.B: “we decompose the input data into data fragments and use time-sharing multiplexing technology to deal with these fragments”; (EN): in combination with HAN, KOLEN, and BETH, the system of HAN now updates its CNN accelerator to use the accelerator techniques of YU which utilizes time-sharing multiplexing, so that based on the first and second priority (as explained above), first and second target data is sent to the accelerator based on such priorities as in BETH)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of HAN, KOLEN, BETH, and YU as explained above. As disclosed by YU, one of ordinary skill would have been motivated to do so the accelerator of YU was experimentally shown to “achieve average 30x speedup.” (p. 1152, section VI).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over HAN in view of KOLEN and further in view of US 20180260690 A1, hereinafter referenced as YOUNG.
Regarding Claim 9
HAN and KOLEN disclose the integrated circuit according to claim 1. However, HAN and KOLEN fail to explicitly teach:
wherein the non real-time response processor comprises a general-purpose matrix processor.
However, in a related field of endeavor (hardware for neural networks, para. 0003), YOUNG teaches:
wherein the non real-time response processor comprises a general-purpose matrix processor. (YOUNG, para. 0038: “the matrix computation unit 312 is a general purpose matrix processor. The special-purpose hardware circuit 300 can use matrix computation unit 312 to perform a matrix transpose operation.”; (EN): in combination with HAN and KOLEN, the system of HAN now has the vision processing DSP 128 of HAN include a general purpose matrix processor for matrix processing as in YOUNG).
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the vehicle system of HAN with KOLEN and YOUNG as explained above. One of ordinary skill would be motivated to do so because YOUNG teaches techniques for “ performing a transpose operation on a matrix without modifying the hardware architecture of the special-purpose hardware circuit. That is, processing delays resulting from performing part of the processing off-chip, in software, or both, are avoided.” (para. 0007). One of ordinary skill would understand that in areas such as image processing and recognition, that matrix processing capabilities would be very useful.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over HAN in view of KOLEN, and further in view of US 20190258924 A1, hereinafter referenced as HAMIDOUCHE.
Regarding Claim 10
HAN and KOLEN disclose the integrated circuit according to claim 1. However, HAN and KOLEN fail to explicitly teach:
wherein parameters in the first neural network model are updated by using a network.
However, in a related field of endeavor (use of deep neural networks for recognition, see para. 0002), HAMIDOUCHE discloses:
wherein parameters in the first neural network model are updated by using a network. (HAMIDOUCHE, para. 0030: “the parameter server 405 determines whether updated parameters for the neural network 404 should be transmitted to remote computing nodes (e.g., 302-304), and transmits the updated parameters as appropriate via the communication network interface 401.”; (EN): in combination with HAN and KOLEN, the neural networks of HAN are now remotely updated over a network using the techniques of HAMIDOUCHE.)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of HAN with KOLEN and HAMIDOUCHE as explained above. As disclosed by HAMIDOUCHE, one of ordinary skill would have been motivated to do so because HAMIDOUCHE teaches an asynchronous approach for updating weights of neural network models that “improves scalability of the training framework because the communication of updates between nodes is not limited by the computational and communication throughput of a single centralized parameter server, and also reduces divergence resulting from the use of stale data.” (para. 0018).
Claims 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over US 20190235488 A1, hereinafter referenced as BETH, in view of US 20180121796 A1, hereinafter referenced as DEISHER, and further in view of US 20200306632 A1, hereinafter referenced as KOLEN.
Regarding Claim 21
BETH teaches:
A wireless communication device, comprising: (BETH, para. 0020: “FIG. 2 depicts exemplary components of vehicle 210A, 210B, which may include a robot 214, various sensors 216, a controller 236, data storage 234, a V2V communication interface 228, a communications interface 226 for communicating with the platform 120 and other external entities via one or more of various types of networks.”
BETH, para. 0022: “The communication interfaces 226, 228 may include a high-speed data transmission connection interface (beyond the on-board connections) including USB, Ethernet, or the like for communicating with one or more other vehicles, the cloud platform, or other entities, as well as include components allowing for different types of data communication on different spectrum, which may include cellular data like LTE, WiFi, and proprietary systems such as iridium satellite connectivity.”
BETH, para. 0073: “However, according to software or hardware engineering practices, the depicted elements and the functions thereof may be implemented on machines through computer executable transitory and/or non-transitory media having a processor capable of executing program instructions stored thereon as a monolithic software structure, as standalone software modules, or as modules that employ external routines, code, services, and so forth, or any combination of these, and all such implementations may be within the scope of the present disclosure. Examples of such machines may include, but may not be limited to, personal digital assistants, laptops, personal computers, mobile phones, other handheld computing devices, medical equipment, wired or wireless communication devices, transducers, chips, calculators, satellites, tablet PCs, electronic books, gadgets, electronic devices, devices having AI, computing devices, networking equipment, servers, routers and the like.”;
Examiner’s Note: BETH discloses a vehicle with 2 communication interfaces that can communicate wirelessly)
first and second wireless transceivers configured to transmit and receive wireless signals; (BETH, para. 0020: “FIG. 2 depicts exemplary components of vehicle 210A, 210B, which may include a robot 214, various sensors 216, a controller 236, data storage 234, a V2V communication interface 228, a communications interface 226 for communicating with the platform 120 and other external entities via one or more of various types of networks.”
BETH, para. 0022: “The communication interfaces 226, 228 may include a high-speed data transmission connection interface (beyond the on-board connections) including USB, Ethernet, or the like for communicating with one or more other vehicles, the cloud platform, or other entities, as well as include components allowing for different types of data communication on different spectrum, which may include cellular data like LTE, WiFi, and proprietary systems such as iridium satellite connectivity.”
Examiner’s Note: the communication interfaces 226 and 228 correspond to the recited “first and second wireless transceivers” because they can include interfaces for communicating over WiFi or cellular which are wireless communication technologies)
at least one processor comprising a first processor and a second processor; (BETH, para. 0062: “The methods and systems described herein may be deployed in part or in whole through a machine that executes computer software, program codes, and/or instructions on a processor. The processor may be part of a server, client, network infrastructure, mobile computing platform, stationary computing platform, or other computing platform. A processor may be any kind of computational or processing device capable of executing program instructions, codes, binary instructions and the like. The processor may be or include a signal processor, digital processor, embedded processor, microprocessor or any variant such as a co-processor (math co-processor, graphic co-processor, communication co-processor and the like) and the like that may directly or indirectly facilitate execution of program code or program instructions stored thereon.”;
BETH, para. 0063: “A processor may include one or more cores that may enhance speed and performance of a multiprocessor. In embodiments, the process may be a dual core processor, quad core processors, other chip-level multiprocessor and the like that combine two or more independent cores (called a die).”;
Examiner’s Note: BETH discloses that the processor can comprise more than one processor, such as a processor having co-processors or a processor having multiple processors on a single die (e.g., multi-core))
a sensor suite coupled to the at least one processor and comprising at least one of:
a pressure sensor; a gyro sensor; a barometric sensor; a magnetic sensor; an acceleration sensor; a distance sensor; a proximity sensor; a fingerprint sensor; a temperature sensor; a touch sensor; an ambient optical sensor; or a bone conductor sensor; (BETH, para. 0021: The sensors may comprise LIDAR 218, IR sensors 220, digital cameras 222, RGB or other video 234, and other sensors such as thermal, stereo, hyper or multispectral sensors, or any other 2D, 3D or other sensor, and the like, for data acquisition related to a mission and navigation of the vehicle.”)
a power management module coupled to the at least one processor; (BETH, para. 0061: “In embodiments, vehicles may have an energy management system. Due to a large power distribution system, the vehicles may have isolated power systems that may operate individually. While there may be built in redundancies and fail safes, a battery management system may be capable of monitoring the state of each battery, and switching its mode (i.e. charging/discharging). This subsystem may be capable of individually monitoring two or more batteries per motor, along with providing the power reserved for the payload and vehicle flight utilities. In embodiments, batteries for a first rotor may be operating in a recharge state that may facilitate collecting air currents with the first rotor acting as a wind-powered generator while batteries for a second rotor may be discharging to power the second rotor.”;
Examiner’s Note: the vehicle’s “energy management system” corresponds to the recited “power management module” and the vehicle controller 236 (corresponding to recited “one processor”) controls all aspects of the vehicle, including the energy management)
a charging management module coupled to the power management module; (BETH, para. 0061: “In embodiments, vehicles may have an energy management system. Due to a large power distribution system, the vehicles may have isolated power systems that may operate individually. While there may be built in redundancies and fail safes, a battery management system may be capable of monitoring the state of each battery, and switching its mode (i.e. charging/discharging). This subsystem may be capable of individually monitoring two or more batteries per motor, along with providing the power reserved for the payload and vehicle flight utilities. In embodiments, batteries for a first rotor may be operating in a recharge state that may facilitate collecting air currents with the first rotor acting as a wind-powered generator while batteries for a second rotor may be discharging to power the second rotor.”;
Examiner’s Note: the logic for charging/discharging the battery corresponds to the recited “charging management module” which is part of the energy management system)
a power source coupled to the power management module; (BETH, para. 0061: “In embodiments, vehicles may have an energy management system. Due to a large power distribution system, the vehicles may have isolated power systems that may operate individually. While there may be built in redundancies and fail safes, a battery management system may be capable of monitoring the state of each battery, and switching its mode (i.e. charging/discharging). This subsystem may be capable of individually monitoring two or more batteries per motor, along with providing the power reserved for the payload and vehicle flight utilities. In embodiments, batteries for a first rotor may be operating in a recharge state that may facilitate collecting air currents with the first rotor acting as a wind-powered generator while batteries for a second rotor may be discharging to power the second rotor.”;
Examiner’s Note: the battery for the vehicle corresponds to the recited “power source”)
memory storing computer instructions that, when executed by the at least one processor, causes the device to: (BETH, para. 0062: “The processor may include memory that stores methods, codes, instructions and programs as described herein and elsewhere. The processor may access a storage medium through an interface that may store methods, codes, and instructions as described herein and elsewhere. The storage medium associated with the processor for storing methods, programs, codes, program instructions or other type of instructions capable of being executed by the computing or processing device may include but may not be limited to one or more of a CD-ROM, DVD, memory, hard disk, flash drive, RAM, ROM, cache and the like.”)
receive data presenting sensor data; (BETH, para. 0003: “Further in the method, the one or more vehicles determined for performing the corresponding mission path and/or segment thereof may be configured with image acquisition devices that maybe operated to acquire digital multi-dimensional data. In embodiments, the digital multi-dimensional data may be acquired digitally, such as from a computer memory and the like. The digital multi-dimensional data may correspond to a flight path and may be analyzed to detect, among other things, incidents relating to the mission which require a change, such as in the flight path and/or in a mission path to satisfy the mission plan.”;
BETH, para. 0021: “The vehicle controller 236 may include a processing element ... for performing vehicle and related data processing, including without limitation neural network real-time data processing. ... The sensors may comprise LIDAR 218, IR sensors 220, digital cameras 222, RGB or other video 234, and other sensors such as thermal, stereo, hyper or multispectral sensors, or any other 2D, 3D or other sensor, and the like, for data acquisition related to a mission and navigation of the vehicle”;
Examiner’s Note: the vehicle controller 236 receives digital camera data from the cameras, for example)
extract a portion of the sensor data; and (BETH, para. 0033: “This information can include tree species, canopy height, canopy area, canopy base, trunk diameter, trunk taper, trunk straightness, and overall foliage. In addition to inventory data, sensors can collect additional data such as health and hydration of the forest.”;
BETH, para. 0035: “Thus, the vehicle may include an intelligent data filtering module 242, which acts to determine which dataset to provide as useful information to the end-user, which is important for the vehicle autonomy perception, and determining which data should be added to various training sets shared amongst vehicles. In embodiments, the data filtering module 242 may compare stored data on the vehicle to the available current or expected bandwidth on available spectrum in order to determine how much and what types of data to send. The vehicle may classify data in order to prioritize the data to transmit.”
(EN): for example, a sensor can be camera 222, which is configured to collect images (corresponding to first sensor digital data) of specific trees, for extracting an image of a first tree (corresponding to the extracted data)) and then the intelligent data filtering module 242 applies logic to extract and send only a portion of the captured camera data, for example, based on rules about bandwidth)
However, BETH fails to explicitly teach:
a user interface coupled to the at least one processor and comprising at least one of:
a speaker; a microphone; a headset jack; or a display;
prompt the first processor to analyze the extracted portion of the sensor data to identify a data type of the extracted portion of the sensor data to determine whether to waken the second processor to further analyze the extracted portion of the sensor data and to enable the second processor to select an associated neural network stored in memory for further processing the extracted portion of the sensor data based on the data type using the selected neural network.
However, in a related field of endeavor (neural network accelerators, see para. 0030), DEISHER teaches:
a user interface coupled to the at least one processor and comprising at least one of:
a speaker; a microphone; a headset jack; or a display; (DEISHER, para. 0287: “In various implementations, display 1420 may include any television type monitor or display, or any smartphone type display. Display 1420 may include, for example, a computer display screen, touch screen display, video monitor, television-like device, and/or a television. Display 1420 may be digital and/or analog. In various implementations, display 1420 may be a holographic display. Also, display 1420 may be a transparent surface that may receive a visual projection. Such projections may convey various forms of information, images, and/or objects. For example, such projections may be a visual overlay for a mobile augmented reality (MAR) application. Under the control of one or more software applications 1416, platform 1402 may display user interface 1422 on display 1420.”)
to determine whether to waken a second processor (DEISHER, para. 0035: “a host CPU (e.g., where the CPU is free to perform other operations or to enter low power sleep state for example)”; (EN): the host CPU, which works in parallel with a “small, flexible, low-power hardware co-processor” is awoken from a sleep state, for example, if additional processing power is needed that the controller 236 of BETH cannot provide)
Before the effective filing date of the present application, it would have been obvious to a person of ordinary skill in the art to combine the teachings of BETH and DEISHER as explained above. As disclosed by DEISHER, one of ordinary skill in the art would understand the benefits of using “on-board chips (such as on an SoC) on the small device or client rather than offloading the ... operations to a remote location (such as a server or the cloud).” (para. 0032). One of ordinary skill would understand that using an on-device chip would result in lower latency.
However, BETH and DEISHER fail to explicitly teach:
prompt the first processor to analyze the extracted portion of the sensor data to identify a data type of the extracted portion of the sensor data to determine whether to waken the second processor to further analyze the extracted portion of the sensor data and to enable the second processor to select an associated neural network stored in memory for further processing the extracted portion of the sensor data based on the data type using the selected neural network
However, in a related field of endeavor (adjusting computing resources, see para. 0042), KOLEN teaches:
prompt the first processor to analyze the extracted portion of the sensor data to identify a data type of the extracted portion of the sensor data to determine whether to waken the second processor to further analyze the extracted portion of the sensor data and to enable the second processor to select an associated neural network stored in memory for further processing the extracted portion of the sensor data based on the data type using the selected neural network. (KOLEN, para. 0062: “Some non-limiting examples of machine learning algorithms that can be used to generate and update the prediction models can include supervised and non-supervised machine learning algorithms, including ... artificial neural network algorithms (such as, for example, Perceptron), deep learning algorithms (such as, for example, Deep Boltzmann Machine), ... and/or other machine learning algorithms.”;
KOLEN, para. 0070: “The user computing system 110 may have varied local computing resources, such as central processing units and architectures, memory, mass storage, graphics processing units, communication network availability and bandwidth, and so forth. Further, the user computing system 110 may include any type of computing system.”
KOLEN, para. 0076: “This control data 156 may identify one or more features or characteristics for which the model generation system 146 is to determine a model.”
KOLEN, para. 0085: “The computing resource adjustment system 118 can determine which prediction model, such as any of models 160A-N, to use based on input data 172 and/or additional identifiers associated with the input data 172. Additionally, the selection of a prediction model 160 may be based on the specific input data 172 provided. The availability of particular types of data as part of the input data 172 can affect the selection of the prediction model 160.”;
Examiner’s Note: KOLEN discloses selecting a specific prediction model (which can be based on neural network algorithms) based on a specific type of input data; the BETH-DEISHER-KOLEN combination now modifies BETH so that a specific type of neural network can be selected based on the input type, e.g., different neural networks for digital camera imagery vs. LIDAR imagery)
Before the effective filing date of the present application, it would have been obvious to a person of ordinary skill in the art to combine teachings of BETH, DEISHER, and KOLEN as explained above. As disclosed by KOLEN, one of ordinary skill in the art would have been motivated to do so in order to predict computing resources so that “unnecessary delays associated with acquiring additional resources can be avoided or reduced.” (para. 0026).
Regarding Claim 22
BETH, DEISHER, and KOLEN disclose the wireless communication device according to claim 21. BETH further teaches:
wherein the at least one processor comprises: the first processor; the second processor; and an accelerator. (BETH, para. 0021: “The vehicle controller 236 may include a processing element such as a GPU, floating point processor, AI accelerator and the like for performing vehicle and related data processing, including without limitation neural network real-time data processing.”
BETH, para. 0062: “The methods and systems described herein may be deployed in part or in whole through a machine that executes computer software, program codes, and/or instructions on a processor. The processor may be part of a server, client, network infrastructure, mobile computing platform, stationary computing platform, or other computing platform. A processor may be any kind of computational or processing device capable of executing program instructions, codes, binary instructions and the like. The processor may be or include a signal processor, digital processor, embedded processor, microprocessor or any variant such as a co-processor (math co-processor, graphic co-processor, communication co-processor and the like) and the like that may directly or indirectly facilitate execution of program code or program instructions stored thereon.”;
BETH, para. 0063: “A processor may include one or more cores that may enhance speed and performance of a multiprocessor. In embodiments, the process may be a dual core processor, quad core processors, other chip-level multiprocessor and the like that combine two or more independent cores (called a die).”;
Examiner’s Note: BETH discloses that the processor can comprise more than one processor, such as a processor having co-processors or a processor having multiple processors on a single die (e.g., multi-core) and that the vehicle controller 236 also includes an AI accelerator)
Regarding Claim 23
BETH, DEISHER, and KOLEN disclose the wireless communication device according to claim 22. However, BETH fails to explicitly teach:
wherein the at least one processor further comprises a non real-time response processor that is configured to select the associated neural network for further processing the extracted portion of the sensor data.
However, in a related field of endeavor (neural network accelerators, see para. 0030), DEISHER teaches:
wherein the at least one processor further comprises a non real-time response processor (DEISHER, para. 0031: “Thus, a number of challenges exist in deploying neural network-based speech recognition (and other pattern recognition capabilities) on wearable devices and other client devices that have relatively small processor areas including: (1) the large amount of compute can lead to long latency on wearable and other small devices, (2) use of the applications processor (such as the central processing unit (CPU)) for these computations requires higher power (and in turn, shortens battery life), and (3) use of the ASR application's processor (or processor shared with other applications) also can degrade the other application's performance.”;
DEISHER, para. 0035: “Thus, the NNA is a small, flexible, low-power hardware co-processor that runs neural network forward propagation in parallel with a host CPU (e.g., where the CPU is free to perform other operations or to enter low power sleep state for example). The CPU may be or may be polling for the completion of the NNA operation. The NNA uses matrices of configurable batching (or grouping) of multiple input vectors (each to provide a layer output) to provide data to the parallel logic and in order to re-use a single fetched input array multiple times which reduces the number of fetches from memory external to the die holding the NNA (or off-chip). Specifically, the use of concurrent (co-)processing by using the parallel logic coupled with memory streaming that uses the input matrices is one of the ways to process the neural network so that the acoustic modeling does not interfere with the application or cache. The details of these advantages as well as other advantages are provided below.”; (EN): the host CPU, which works in parallel with a “small, flexible, low-power hardware co-processor” corresponds to the recited “non real-time response processor”; the BETH-DEISHER-KOLEN combination now adds the host CPU of DEISHER to the system of BETH, such that controller 236 of BETH (the “real-time response processor”) can offload computations that require more power to the CPU as described by DEISHER)
Before the effective filing date of the present application, it would have been obvious to a person of ordinary skill in the art to combine the teachings of BETH, DEISHER, and KOLEN as explained above. As disclosed by DEISHER, one of ordinary skill in the art would understand the benefits of using “on-board chips (such as on an SoC) on the small device or client rather than offloading the ... operations to a remote location (such as a server or the cloud).” (para. 0032). One of ordinary skill would understand that using an on-device chip would result in lower latency.
However, BETH and DEISHER fail to explicitly teach:
that is configured to select the associated neural network for further processing the extracted portion of the sensor data.
However, in a related field of endeavor (adjusting computing resources, see para. 0042), KOLEN teaches:
that is configured to select the associated neural network for further processing the extracted portion of the sensor data. (KOLEN, para. 0062: “Some non-limiting examples of machine learning algorithms that can be used to generate and update the prediction models can include supervised and non-supervised machine learning algorithms, including ... artificial neural network algorithms (such as, for example, Perceptron), deep learning algorithms (such as, for example, Deep Boltzmann Machine), ... and/or other machine learning algorithms.”;
KOLEN, para. 0085: “The computing resource adjustment system 118 can determine which prediction model, such as any of models 160A-N, to use based on input data 172 and/or additional identifiers associated with the input data 172. Additionally, the selection of a prediction model 160 may be based on the specific input data 172 provided. The availability of particular types of data as part of the input data 172 can affect the selection of the prediction model 160.”;
Examiner’s Note: KOLEN discloses selecting a specific prediction model (which can be based on neural network algorithms) based on a specific type of input data; the BETH-DEISHER-KOLEN combination now modifies BETH so that a specific type of neural network can be selected based on the input type, e.g., different neural networks for digital camera imagery vs. LIDAR imagery)
Before the effective filing date of the present application, it would have been obvious to a person of ordinary skill in the art to combine the teachings of BETH, DEISHER, and KOLEN as explained above. As disclosed by KOLEN, one of ordinary skill in the art would have been motivated to do so in order to predict computing resources so that “unnecessary delays associated with acquiring additional resources can be avoided or reduced.” (para. 0026).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20190043488 A1 (Bocklet). “To resolve these issues, the present method and system use a neural network keyphrase detection decoder (or keyphrase detection or Wake on Voice (WoV) engine or just keyphrase decoder) that can be used in a start-to-finish neural network keyphrase detection system that limits or eliminates the need for a DSP and instead can be processed on highly efficient neural network accelerators in an autonomous, stand-alone manner.” (para. 0029).
US 20170242806 A1 (Solbach). “Once triggering data 310 has been detected, sensor processor 172 may make a decision to send a command, such as an interrupt, to wake host processor 110 from a low-power mode so that it enters a higher power mode where it is capable of performing more tasks.” (para. 0064).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL C LEE whose telephone number is (571)272-4933. The examiner can normally be reached M-F 12:00 pm - 8:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at 571-272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL C. LEE/Examiner, Art Unit 2128
Prosecution Insights