DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the
first inventor to file provisions of the AIA .
Priority
2. Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged. The instant application claims priority to U.S. provisional application No. 63/245,148 (hereinafter; “provisional application ‘148”) filed 9/16/2021. The instant application is also a continuation in part (CIP) of U.S. Application No. 16/933,859 (hereinafter; “the parent application”), filed 7/20/2020, which claims priority to U.S. provisional application No. 63/025,580 (hereinafter; “provisional application ‘580”) filed 05/15/2020, U.S. provisional application No. 62/941,646 (hereinafter; “provisional application ‘646”) filed 11/27/2019, and U.S. provisional application No. 62/876,219 (hereinafter; “provisional application ‘219”) filed 07/19/2019.
The later-filed application must be an application for a patent for an invention which is also disclosed in the prior application (the parent or original or provisional application). The disclosure of the invention in the parent application and in the later-filed application must be sufficient to comply with the requirements of 35 U.S.C. 112(a) or the first paragraph of pre-AIA 35 U.S.C. 112, except for the best mode requirement. See Transco Products, Inc. v. Performance Contracting, Inc., 38 F.3d 551, 32 USPQ2d 1077 (Fed. Cir. 1994).
As discussed below, the claimed features of dependent claims 5, 6, 8, 15, 16 and 18 are not supported by provisional application ‘219. Therefore, claims 5, 6, 8, 15, 16 and 18 do not enjoy the priority benefit of the provisional application ‘219 filed 07/19/2019.
Claims 5 and 15 recite “wherein the serial communication link is a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI)”. The as-filed specification of provisional application ‘219 and provisional application ‘148 are silent regarding a CSI interface of the MIPI standard. Thus, the discussed prior applications fail to provide adequate support of enablement for at least the above-noted element of claims 5 and 15. Based on their respective dependencies from claims 5 and 15, the specifications of the prior applications also fail to provide adequate support or enablement for dependent claims 6 and 16, respectively.
Provisional application ‘580 appears to provide adequate support and enablement for the elements of claims 5, 6, 15 and 16, as it discloses “Uses MIPI-CSI bus to move data through CNN Processor”. One of ordinary skill in the art would determine the elements of claims 6 and 16 of “MIPI packet comprises the payload portion and a metadata portion” and “send the portion of the preselected data in the metadata portion of the MIPI packet”, are inherently included in the provisional application ‘580 by the disclosed MIPI-CSI bus used to ‘move data through CNN processor’. Therefore, claims 5, 6, 15 and 16 enjoy the priority benefit of provisional application ‘580 filed 05/15/2020.
Claims 8 and 18 recite “wherein the image sensor comprises a camera;”. The as-filed specification of provisional application ‘148 is silent regarding an image sensor comprising a camera. Thus, the discussed prior application fails to provide adequate support or enablement for at least the above-noted element of claims 8 and 18.
Provisional application ‘646 filed 11/27/2019, appears to provide adequate support and enablement for the elements of claims 8 and 18, as it discloses “When a picture is taken, a digital camera initially produces a raw Bayer pixel array from the image sensor”. Therefore, claims 8 and 18 enjoy the priority benefit of provisional application ‘646 filed 11/27/2019.
The specification of the provisional application ‘219 appears to provide adequate support and enablement for the elements of claims 1-4, 7, 9-14, 17 and 19-20, as currently written. Examiner will consider if the parent application and the other, above-noted provisional applications support each dependent claim if a rejection would need to rely upon an intervening reference between the actual filing date of the instant application, 09/14/2022, and the 07/19/2019 filing of the provisional application ‘219. As such, each of the mentioned claims will receive benefit of the earliest filing date above for which a continuous chain of support can be established for the entirety of the claim.
Each claim will receive benefit of the earliest filing date above for which a continuous chain of support can be established for the entirety of the claim. As discussed above, the effective filing date for claims 1-4, 7, 9-14, 17 and 19-20 of the instant application is the filing date of provisional application ‘219, 07/19/2019.
Information Disclosure Statement
3. The information disclosure statement (IDS) submitted on 4/24/2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
4. The drawings are objected to as failing to comply with 37 CFR 1.84(p)(3) because Figures 13, 16-18, 20, 22 and 24 include letters which do not measure at least .32 cm. (1/8 inch) in height (see, e.g., most of the dimension values in Figs. 16-18, 20, 22 and 24). See MPEP 507 (A) and 37 CFR 1.84(p)(3): Numbers, letters, and reference characters must measure at least .32 cm. (1/8 inch) in height.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because:
Reference character “606” has been used to designate both “a register-module interface", in paragraph [0091] and “Config-Reg [52]x8” in Fig. 6.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference characters not mentioned in the description:
Reference No. 3518 in Fig. 35 is not mentioned in the disclosure (see, e.g., first chip test paragraph [00178]).
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
5. The disclosure is objected to because of the following informalities:
Reference No. 3518 in Fig. 35 is not mentioned in the disclosure (see, e.g., first chip test paragraph [00178]).
Reference character “606” has been used to designate both “a register-module interface", in paragraph [0091] and “Config-Reg [52]x8” in Fig. 6.
Appropriate correction is required.
Claim Interpretation
6. The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
7. The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
the first processing element is configured to: generate preselected data in claim 11.
the first processing element is further configured to: (1) send… in claim 12.
the first processing element is further configured to: send… in claim 13.
the first processing element is configured to send… in claim 16.
the second processing element by being further configured to send in claim 16.
the second processing element are configured to perform at least one function of a preselected CNN in claim 19.
the first processing element and the second processing element… and configured to work together to implement a preselected CNN in claim 20.
Regarding claim 11 and the above-noted three-prong test, the recited first processing element is a generic placeholder, to generate… is functional language, and, and there’s no recitation of sufficient structure in claim 11 to perform the generating.
Regarding claims 12, 13 and 16 and the above-noted three-prong test, the recited first processing element is a generic placeholder, to send… is functional language, and there’s no recitation of sufficient structure in claims 12, 13, or 16 to perform the sending.
Regarding claims 12, 13 and 16 and the above-noted three-prong test, the recited first processing element is a generic placeholder, to send… is functional language, and there’s no recitation of sufficient structure in claims 12, 13, or 16 to perform the sending.
Regarding claims 16 and 19 and the above-noted three-prong test, the recited second processing element is a generic placeholder, to send and perform at least one function of a preselected CNN… are examples of functional language, and there’s no recitation of sufficient structure in claim 16 or 19 to perform the sending or performing functions of a preselected CNN.
Regarding claim 20 and the above-noted three-prong test, the recited first processing element and the second processing element are generic placeholders, to implement a preselected CNN … is functional language, and, and there’s no recitation of sufficient structure in claim 11 to perform the implementing.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
A review of the specification shows that the corresponding structure is described in the specification for the recited processing elements recited in claims 11, 12, 13, 16, 19 and 20. Specifically, claim 11 states “A processor dedicated to implementing a convolution neural network (CNN), comprising: a first processing element… and a second processing element”, meaning the processor in claim 11 contains the aforementioned processing elements being interpreted under 112(f).
Paragraph [0156] discloses “each of the processing elements (e.g., one PFU implementing multiple PFUs) may be implemented in one integrated circuit (e.g., a chip) and communication across the chips can be performed over MIPI”. Paragraph [00202] discloses “the first processing element and the second processing element are implemented in separate integrated circuits and configured to work together to implement a preselected CNN (e.g., the target CNN with a CNN graph used by the compiler to fit the target CNN to the configurable/scalable CNN processor architecture)” The processing elements are said be implemented on a physical integrated circuit/chip, which is interpreted to provide sufficient structure for said processing elements described in the claims.
8. If applicant does not intend to have these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
9. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
10. Claims 1-20 are rejected under 35 U.S.C. 101 because the claims are directed towards an
abstract idea without significantly more.
Regarding Independent Claim 1:
Step 1: The claim is directed to a method, corresponding to a process, which is one of the statutory categories.
Step 2A, Prong 1: The following limitation is directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind or with pen and paper (including an observation, evaluation, judgement, or opinion).
generating preselected data to be communicated using a serial communication link between a first processing element and a second processing element within the CNN processor;
Regarding the “generating preselected data”, this generating can be associated with the mental process of observation and evaluation of preselected data. Given a sufficiently small enough dataset of preselected values, but for the recitation of generic computer components (i.e., using a serial communication link between a first processing element and a second processing element within the CNN processor), nothing in the claim prohibits this process from being performed mentally or with a pen and paper.
If the claim limitations, under their broadest reasonable interpretations (BRIs), cover performance of the limitations in the mind but for the recitation of generic computer components (i.e., the generically-recited “serial communication link between a first processing element and a second processing element within the CNN processor”), then they fall within the “Mental Processes” grouping of abstract ideas. Accordingly, claim 1 recites an abstract idea.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements add insignificant extra-solution activities (necessary data gathering and data storage) to the judicial exception [see MPEP 2106.05(g)].
A method for communicating between processing elements within a processor dedicated to implementing a convolution neural network (CNN), comprising:
receiving image data from an image sensor,
sending, via the serial communication link, a first row of the first image data from the first processing element to the second processing element;
sending, via the serial communication link, a first row of the first image data from the first processing element to the second processing element;
and sending, via the serial communication link, a second row of the first image data from the first processing element to the second processing element
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the image data. Therefore, the additional element does not integrate the abstract ideas into a practical application.
the image data comprising a first image data comprising multiple rows of data;
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are directed to receiving, communicating, and storing data as insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
A method for communicating between processing elements within a processor dedicated to implementing a convolution neural network (CNN), comprising:
receiving image data from an image sensor,
sending, via the serial communication link, a first row of the first image data from the first processing element to the second processing element;
sending, via the serial communication link, a first row of the first image data from the first processing element to the second processing element;
and sending, via the serial communication link, a second row of the first image data from the first processing element to the second processing element
Regarding Claim 2:
Step 1: The claim is directed to the process of claim 1.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 1.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements add insignificant extra-solution activities (necessary data gathering and data storage) to the judicial exception [see MPEP 2106.05(g)].
(1) sending, via the serial communication link, another portion of the preselected data from the first processing element to the second processing element;
(2) sending, via the serial communication link, another row of the first image data from the first processing element to the second processing element;
and repeating (1) and (2) until the entire first image data has been sent or the entire preselected data has been sent
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are directed to receiving or transmitting data over a network. The courts (as per Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 / buySAFE, Inc. v. Google, Inc., 765 F.3d at 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014)) have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(d) II.] and therefore fails to amount to significantly more than the judicial exception.
(1) sending, via the serial communication link, another portion of the preselected data from the first processing element to the second processing element;
(2) sending, via the serial communication link, another row of the first image data from the first processing element to the second processing element;
The following limitation can be characterized as insignificant extra solution activity that is well understood routine and conventional. See MPEP 2106.05(g) and MPEP 2106.05(d)(II) example (ii) provides that performing repetitive calculations has been understood by the courts to be well-understood, routine and conventional.
and repeating (1) and (2) until the entire first image data has been sent or the entire preselected data has been sent
Regarding Claim 3:
Step 1: The claim is directed to the process of claim 2.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 2.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the image data and process of sending the generated data via the serial communication link. Therefore, the additional element does not integrate the abstract ideas into a practical application.
wherein the image data further comprises a second image data;
The following additional elements add insignificant extra-solution activities (necessary data gathering and data storage) to the judicial exception [see MPEP 2106.05(g)].
and wherein the method further comprises: sending, via the serial communication link, another portion of the preselected data from the first processing element to the second processing element;
and sending, via the serial communication link, the second image data from the first processing element to the second processing element
wherein (1) and (2) were repeated until the entire first image data was sent;
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are directed to receiving or transmitting data over a network. The courts (as per Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 / buySAFE, Inc. v. Google, Inc., 765 F.3d at 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014)) have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(d) II.] and therefore fails to amount to significantly more than the judicial exception.
and wherein the method further comprises: sending, via the serial communication link, another portion of the preselected data from the first processing element to the second processing element;
and sending, via the serial communication link, the second image data from the first processing element to the second processing element
The following limitation can be characterized as insignificant extra solution activity that is well understood routine and conventional. See MPEP 2106.05(g) and MPEP 2106.05(d)(II) example (ii) provides that performing repetitive calculations has been understood by the courts to be well-understood, routine and conventional.
wherein (1) and (2) were repeated until the entire first image data was sent;
Regarding Claim 4:
Step 1: The claim is directed to the process of claim 1.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 1.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the preselected data transmitted between processing elements. Therefore, the additional element does not integrate the abstract ideas into a practical application.
wherein the preselected data comprises a tensor used by at least one of the first processing element or the second processing element to perform a CNN function
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional limitation amounts to no more than using generic computer components to implement the exception. Limiting the abstract idea to a particular technological context or field of use, does not render the claim patent eligible. Therefore, claim 14 is not patent eligible.
Regarding Claim 5:
Step 1: The claim is directed to the process of claim 1.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 1.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the serial communication link used for transmitting the preselected data. Therefore, the additional element does not integrate the abstract ideas into a practical application.
wherein the serial communication link is a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional limitation amounts to no more than using generic computer components to implement the exception. Limiting the abstract idea to a particular technological context or field of use, does not render the claim patent eligible. Therefore, claim 5 is not patent eligible.
Regarding Claim 6:
Step 1: The claim is directed to the process of claim 1.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 1.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements add insignificant extra-solution activities (necessary data gathering and data storage) to the judicial exception [see MPEP 2106.05(g)].
wherein the sending, via the serial communication link, the first row of the first image data from the first processing element to the second processing element comprises sending the first row of the first image data in a payload portion of a MIPI packet,
and wherein the sending, via the serial communication link, the portion of the preselected data comprises sending the portion of the preselected data in the metadata portion of the MIPI packet
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the image data sent via the serial communication link. Therefore, the additional element does not integrate the abstract ideas into a practical application.
wherein the MIPI packet comprises the payload portion and a metadata portion;
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are directed to receiving or transmitting data over a network. The courts (as per Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 / buySAFE, Inc. v. Google, Inc., 765 F.3d at 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014)) have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(d) II.] and therefore fails to amount to significantly more than the judicial exception.
wherein the sending, via the serial communication link, the first row of the first image data from the first processing element to the second processing element comprises sending the first row of the first image data in a payload portion of a MIPI packet,
and wherein the sending, via the serial communication link, the portion of the preselected data comprises sending the portion of the preselected data in the metadata portion of the MIPI packet
Regarding Claim 7:
Step 1: The claim is directed to the process of claim 1.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 1.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the serial communication link used to transmit data between processing elements. Therefore, the additional element does not integrate the abstract ideas into a practical application.
wherein the serial communication link is implemented using at least one of a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI), a High-Definition Multimedia Interface (HDMI), or a Serializer/Deserializer (SerDes)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional limitation amounts to no more than using generic computer components to implement the exception. Limiting the abstract idea to a particular technological context or field of use, does not render the claim patent eligible. Therefore, claim 7 is not patent eligible.
Regarding Claim 8:
Step 1: The claim is directed to the process of claim 1.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 1.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements do not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the image data and components used for the generation of the data. Therefore, the additional elements do not integrate the abstract ideas into a practical application.
wherein the image sensor comprises a camera;
and wherein the image data comprises video data comprising multiple frames
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional limitations amount to no more than using generic computer components to implement the exception. Limiting the abstract idea to a particular technological context or field of use, does not render the claim patent eligible. Therefore, claim 8 is not patent eligible.
Regarding Claim 9:
Step 1: The claim is directed to the process of claim 1.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 1.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional element is adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
wherein each of the first processing element and the second processing element are configured to perform at least one function of a preselected CNN,
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the function performed by the processing elements on the generated data. Therefore, the additional element does not integrate the abstract ideas into a practical application.
the at least one function comprising at least one of convolution, batch normalization, pooling, or activation
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional limitations amount to no more than using generic computer components to implement the exception. Implementing the abstract idea by merely applying it using generic computer components, without more, does not amount to an inventive concept. Further, limiting the abstract idea to a particular technological context or field of use, does not render the claim patent eligible. Therefore, claim 9 is not patent eligible.
Regarding Claim 10:
Step 1: The claim is directed to the process of claim 1.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 1.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements do not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the processing elements used in the generation and transmission of data. Therefore, the additional elements do not integrate the abstract ideas into a practical application.
wherein the first processing element and the second processing element are implemented in separate integrated circuits
The following additional element is adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
and configured to work together to implement a preselected CNN
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional limitations amount to no more than using generic computer components to implement the exception. Implementing the abstract idea by merely applying it using generic computer components, without more, does not amount to an inventive concept. Further, limiting the abstract idea to a particular technological context or field of use, does not render the claim patent eligible. Therefore, claim 10 is not patent eligible.
Regarding Independent Claim 11:
Step 1: The claim is directed to a processor, corresponding to an article of manufacture, which is one of the statutory categories.
Step 2A, Prong 1: The following limitation is directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind or with pen and paper (including an observation, evaluation, judgement, or opinion).
wherein the first processing element is configured to1: generate preselected data to be communicated using the serial communication link;
Regarding the “the first processing element is configured to: generate preselected data”, this generating can be associated with the mental process of observation and evaluation of preselected data. Given a sufficiently small enough dataset of preselected values, but for the recitation of generic computer components (i.e., using a serial communication link between a first processing element and a second processing element within the CNN processor), nothing in the claim prohibits this process from being performed mentally or with a pen and paper.
If the claim limitations, under their broadest reasonable interpretations (BRIs), cover performance of the limitations in the mind but for the recitation of generic computer components (i.e., the generically-recited “first processing element coupled to an image sensor; and a second processing element coupled to the first processing element via a serial communication link”), then they fall within the “Mental Processes” grouping of abstract ideas. Accordingly, claim 11 recites an abstract idea.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional element is adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
A processor dedicated to implementing a convolution neural network (CNN), comprising:
The following additional elements add insignificant extra-solution activities (necessary data gathering and data storage) to the judicial exception [see MPEP 2106.05(g)].
receive image data from the image sensor,
send, via the serial communication link, a portion of the preselected data to the second processing element;
and send, via the serial communication link, a second row of the first image data to the second processing element
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the image data. Therefore, the additional element does not integrate the abstract ideas into a practical application.
a first processing element coupled to an image sensor;
and a second processing element coupled to the first processing element via a serial communication link;
the image data comprising a first image data comprising multiple rows of data;
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are directed to receiving, communicating, and storing data as insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
receive image data from the image sensor,
send, via the serial communication link, a portion of the preselected data to the second processing element;
and send, via the serial communication link, a second row of the first image data to the second processing element
Regarding Claim 12:
Step 1: The claim is directed to the article of manufacture of claim 11.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 11.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements add insignificant extra-solution activities (necessary data gathering and data storage) to the judicial exception [see MPEP 2106.05(g)].
wherein the first processing element is further configured to: (1) send, via the serial communication link, another portion of the preselected data to the second processing element;
(2) send, via the serial communication link, another row of the first image data to the second processing element;
and repeat (1) and (2) until the entire first image data has been sent or the entire preselected data has been sent
The following additional elements are directed to receiving, communicating, and storing data as insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
wherein the first processing element is further configured to: (1) send, via the serial communication link, another portion of the preselected data to the second processing element;
(2) send, via the serial communication link, another row of the first image data to the second processing element;
The following limitation can be characterized as insignificant extra solution activity that is well understood routine and conventional. See MPEP 2106.05(g) and MPEP 2106.05(d)(II) example (ii) provides that performing repetitive calculations has been understood by the courts to be well-understood, routine and conventional.
and repeat (1) and (2) until the entire first image data has been sent or the entire preselected data has been sent
Regarding Claim 13:
Step 1: The claim is directed to the article of manufacture of claim 12.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 2.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the image data and process of sending the generated data via the serial communication link. Therefore, the additional element does not integrate the abstract ideas into a practical application.
wherein the image data further comprises a second image data;
The following additional elements add insignificant extra-solution activities (necessary data gathering and data storage) to the judicial exception [see MPEP 2106.05(g)].
and wherein the first processing element is further configured to: send, via the serial communication link, another portion of the preselected data to the second processing element;
and send, via the serial communication link, the second image data to the second processing element
wherein (1) and (2) were repeated until the entire first image data was sent;
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are directed to receiving, communicating, and storing data as insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
and wherein the first processing element is further configured to: send, via the serial communication link, another portion of the preselected data to the second processing element;
and send, via the serial communication link, the second image data to the second processing element
The following limitation can be characterized as insignificant extra solution activity that is well understood routine and conventional. See MPEP 2106.05(g) and MPEP 2106.05(d)(II) example (ii) provides that performing repetitive calculations has been understood by the courts to be well-understood, routine and conventional.
wherein (1) and (2) were repeated until the entire first image data was sent;
Regarding Claim 14:
Step 1: The claim is directed to the article of manufacture of claim 11.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 11.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the preselected data transmitted between processing elements. Therefore, the additional element does not integrate the abstract ideas into a practical application.
wherein the preselected data comprises a tensor used by at least one of the first processing element or the second processing element to perform a CNN function
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional limitation amounts to no more than using generic computer components to implement the exception. Limiting the abstract idea to a particular technological context or field of use, does not render the claim patent eligible. Therefore, claim 14 is not patent eligible.
Regarding Claim 15:
Step 1: The claim is directed to the article of manufacture of claim 11.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 11.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the serial communication link used for transmitting the preselected data. Therefore, the additional element does not integrate the abstract ideas into a practical application.
wherein the serial communication link is a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional limitation amounts to no more than using generic computer components to implement the exception. Limiting the abstract idea to a particular technological context or field of use, does not render the claim patent eligible. Therefore, claim 15 is not patent eligible.
Regarding Claim 16:
Step 1: The claim is directed to the article of manufacture of claim 11.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 11.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements add insignificant extra-solution activities (necessary data gathering and data storage) to the judicial exception [see MPEP 2106.05(g)].
wherein the first processing element is configured to send, via the serial communication link, a first row of the first image data to the second processing element by being further configured to send the first row of the first image data in a payload portion of a MIPI packet,
and wherein the first processing element is configured to send, via the serial communication link, the portion of the preselected data by being further configured to send the portion of the preselected data in the metadata portion of the MIPI packet
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the image data sent via the serial communication link. Therefore, the additional element does not integrate the abstract ideas into a practical application.
wherein the MIPI packet comprises the payload portion and a metadata portion;
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are directed to receiving or transmitting data over a network. The courts (as per Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 / buySAFE, Inc. v. Google, Inc., 765 F.3d at 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014)) have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(d) II.] and therefore fails to amount to significantly more than the judicial exception.
wherein the first processing element is configured to send, via the serial communication link, a first row of the first image data to the second processing element by being further configured to send the first row of the first image data in a payload portion of a MIPI packet,
and wherein the first processing element is configured to send, via the serial communication link, the portion of the preselected data by being further configured to send the portion of the preselected data in the metadata portion of the MIPI packet
Regarding Claim 17:
Step 1: The claim is directed to the article of manufacture of claim 11.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 11.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the serial communication link used to transmit data between processing elements. Therefore, the additional element does not integrate the abstract ideas into a practical application.
wherein the serial communication link is implemented using at least one of a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI), a High-Definition Multimedia Interface (HDMI), or a Serializer/Deserializer (SerDes)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional limitation amounts to no more than using generic computer components to implement the exception. Limiting the abstract idea to a particular technological context or field of use, does not render the claim patent eligible. Therefore, claim 17 is not patent eligible.
Regarding Claim 18:
Step 1: The claim is directed to the article of manufacture of claim 11.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 11.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements do not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the image data and components used for the generation of the data. Therefore, the additional elements do not integrate the abstract ideas into a practical application.
wherein the image sensor comprises a camera;
and wherein the image data comprises video data comprising multiple frames
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional limitations amount to no more than using generic computer components to implement the exception. Limiting the abstract idea to a particular technological context or field of use, does not render the claim patent eligible. Therefore, claim 18 is not patent eligible.
Regarding Claim 19:
Step 1: The claim is directed to the article of manufacture of claim 11.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 11.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional element is adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
wherein each of the first processing element and the second processing element are configured to perform at least one function of a preselected CNN,
The following additional element does not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the function performed by the processing elements on the generated data. Therefore, the additional element does not integrate the abstract ideas into a practical application.
the at least one function comprising at least one of convolution, batch normalization, pooling, or activation
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional limitations amount to no more than using generic computer components to implement the exception. Implementing the abstract idea by merely applying it using generic computer components, without more, does not amount to an inventive concept. Further, limiting the abstract idea to a particular technological context or field of use, does not render the claim patent eligible. Therefore, claim 19 is not patent eligible.
Regarding Claim 20:
Step 1: The claim is directed to the article of manufacture of claim 11.
Step 2A, Prong 1: The claim recites the same abstract ideas as in claim 11.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements do not meaningfully limit the judicial exception [see MPEP 2106.05(e)]. The claim simply recites additional information regarding the characteristics of the processing elements used in the generation and transmission of data. Therefore, the additional elements do not integrate the abstract ideas into a practical application.
wherein the first processing element and the second processing element are implemented in separate integrated circuits
The following additional element is adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
and configured to work together to implement a preselected CNN
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional limitations amount to no more than using generic computer components to implement the exception. Implementing the abstract idea by merely applying it using generic computer components, without more, does not amount to an inventive concept. Limiting the abstract idea to a particular technological context or field of use, does not render the claim patent eligible. Therefore, claim 20 is not patent eligible.
Claim Rejections - 35 USC § 103
11. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
12. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
13. Claims 1-4, 8-14 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yang (US 20190318226 A1; hereinafter Yang) in view of Shoaib (US 20160267111 A1; hereinafter Shoaib).
Regarding Independent Claim 1, Yang teaches A method for communicating between processing elements2 within a processor dedicated to implementing a convolution neural network (CNN), comprising (see, e.g., Yang paragraph [0036]: “The integrated circuit 100 is implemented as a digital semi-conductor chip and contains a CNN processing engine controller 110, and one or more neural networks (CNN) processing engines 102 operatively coupled to at least one input/output (I/O) data bus 120. Controller 110 is configured to control various operations of the CNN processing engines 102 for extracting features out of an input image based on an image processing technique by performing multiple layers of 3×3 convolutions with rectifications or other nonlinear operations”):
generating preselected data to be communicated using a serial communication link between a first processing element and a second processing element within the CNN processor (see, e.g., Yang paragraph [0085]: “The entire set of trained coefficients or weights are pre-configured to the CNN based integrated circuit as a feature extractor for a particular data format (e.g., imagery data, voice spectrum, fingerprint, palm-print, optical character recognition (OCR), etc.). In general, there are many convolutional layers with many filters in each layer” [i.e., generation of preconfigured coefficients/weight data during training of a CNN is functionally equivalent to preselected data] and paragraph [0088]: “input buffer of a CNN based IC is configured for storing the output from a previous CNN based IC in each group of serially-connected CNN based ICs. In order to facilitate connections in series and in parallel, additional circuitry is added to the CNN based integrated circuit 100. One example bus for network bus 1995 is Universal Serial Bus (USB). Similarly, network buses 1901, 1902, . . . , 1908 can also be USB” [i.e., serial connection between CNN based ICs (processing elements) of the processing system 1900]);
the image data comprising a first image data comprising multiple rows of data (see, e.g., Yang paragraph [0070]: “Imagery data received from the I/O data bus are in form of M×M pixels of imagery data in consecutive blocks… The top and the bottom rows and four corners of the received M×M pixels of imagery data are stored into respective buffers of corresponding blocks based on the geometry of the input image”);
sending, via the serial communication link, a first row of the first image data from the first processing element to the second processing element (see, e.g., Yang paragraph [0087]: “CNN based ICs are connected in series with each to store filter coefficients of a portion of the deep learning model. The large deep learning model can thus be processed with a number of CNN based ICs connected in series” [i.e., the coefficients (preselected data) of the neural network are distributed across a series of ICs (processing elements)] and paragraph [0088]: “Referring now to FIG. 19, it is shown an example deep learning image processing system 1900. Deep learning image processing system 1900 contains multiple groups (e.g., group-1 1910, group-2 1920, . . . , group-n 1980) of CNN based ICs. Groups are connected in parallel via a network bus 1995… input buffer of a CNN based IC is configured for storing the output from a previous CNN based IC in each group of serially-connected CNN based ICs”);
sending, via the serial communication link, a portion of the preselected data from the first processing element to the second processing element (see, e.g., Yang paragraph [0091]: “Four CNN based ICs 2101-2104 are configured to process respective portions 2011-2014 such that a larger deep learning model can be processed. In other words, a large deep leaning model is processed sequentially by a cascade of CNN based ICs connected in series”);
and sending, via the serial communication link, a second row of the first image data from the first processing element to the second processing element (see, e.g., Yang paragraph [0092]: “An input data is partitioned into smaller subsections such that each CNN based IC can handle. In this example, the input data 2200 is partitioned… each subsection 2211 includes at least one overlapped row and column at it border with other neighboring subsection. In one embodiment, each overlapped column and row contains one-pixel thickness… each overlapped column and row contains one-pixel thickness. Four subsections 2201-2204 are processed by an example deep learning image processing system 2300 shown in FIG. 23” [i.e., multiple rows of pixel data (image input data) are sent through ICs connected in series for CNN operations]).
Although Yang substantially teaches the claimed invention, Yang fails to explicitly teach
the limitation receiving image data from an image sensor,
In the same field, analogous art Shoaib teaches receiving image data from an image sensor (see, e.g., Shoaib paragraph [0028]: “A sensor module 201 includes a sensor 202 and a camera serial interface (CSI) 204 and/or a video interface (VI) 206 coupled to the sensor 202. In some examples, the sensor 202 is configured to capture one or more raw images 228 or frames of video, which are transmitted through the CSI 204 and/or VI 206 and transmitted to or placed onto a first frame bus (e.g., frame bus) 224”).
Yang and Shoaib are analogous art because they are both directed to image processing and classification (see, e.g., Yang, paragraph [0079] and Shoaib, paragraph [0070]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang to incorporate the teachings of Shoaib to utilize an image sensor for receiving image data. Doing so would have allowed Yang to use Shoaib’s method in order to “generate a processed image”, as suggested by Shoaib (see, e.g., Shoaib, paragraph [0017]).
Regarding Claim 2, as discussed above, Yang in view of Shoaib teaches the method of claim 1.
Yang further teaches (1) sending, via the serial communication link, another portion of the preselected data from the first processing element to the second processing element (see, e.g., Yang paragraph [0087]: “CNN based ICs are connected in series with each to store filter coefficients of a portion of the deep learning model. The large deep learning model can thus be processed with a number of CNN based ICs connected in series” [i.e., the coefficients (preselected data) of the neural network are distributed across a series of ICs (processing elements)]);
(2) sending, via the serial communication link, another row of the first image data from the first processing element to the second processing element (see, e.g., Yang paragraph [0092]: “In order to ensure proper processing between subsections, each subsection 2211 includes at least one overlapped row and column at it border with other neighboring subsection. In one embodiment, each overlapped column and row contains one-pixel thickness. Four subsections 2201-2204 are processed by an example deep learning image processing system 2300 shown in FIG. 23” [i.e., overlapped row and columns at each subsection suggests a subsequent processing of image data via the serial connection of ICs (processing elements)]);
and repeating (1) and (2) until the entire first image data has been sent or the entire preselected data has been sent (see, e.g., Yang paragraph [0077]: “The convolution-to-pooling procedures are repeated for several layers and finally connected to a Fully-connected Networks (FCN) 1560” and paragraph [0091]: “Four CNN based ICs 2101-2104 are configured to process respective portions 2011-2014 such that a larger deep learning model can be processed. In other words, a large deep leaning model is processed sequentially by a cascade of CNN based ICs connected in series” [i.e., the processing of the larger deep learning model in a sequential cascade of CNN units implies a continuous process that iteratively produces the entire image]).
Regarding Claim 3, as discussed above, Yang in view of Shoaib teaches the method of claim 2.
Yang further teaches wherein (1) and (2) were repeated until the entire first image data was sent (see, e.g., Yang paragraph [0077]: “The convolution-to-pooling procedures are repeated for several layers and finally connected to a Fully-connected Networks (FCN) 1560” and paragraph [0091]: “Four CNN based ICs 2101-2104 are configured to process respective portions 2011-2014 such that a larger deep learning model can be processed. In other words, a large deep leaning model is processed sequentially by a cascade of CNN based ICs connected in series” [i.e., the processing of the larger deep learning model in a sequential cascade of CNN units implies a continuous process that iteratively produces the entire image]);
and wherein the method further comprises: sending, via the serial communication link, another portion of the preselected data from the first processing element to the second processing element (see, e.g., Yang paragraph [0088]: “Referring now to FIG. 19, it is shown an example deep learning image processing system 1900. Deep learning image processing system 1900 contains multiple groups (e.g., group-1 1910, group-2 1920, . . . , group-n 1980) of CNN based ICs. Groups are connected in parallel via a network bus 1995… input buffer of a CNN based IC is configured for storing the output from a previous CNN based IC in each group of serially-connected CNN based ICs”);
and sending, via the serial communication link (see, e.g., Yang paragraph [0088]: “Similarly, input buffer of a CNN based IC is configured for storing the output from a previous CNN based IC in each group of serially-connected CNN based ICs. In order to facilitate connections in series and in parallel, additional circuitry is added to the CNN based integrated circuit 100. One example bus for network bus 1995 is Universal Serial Bus (USB). Similarly, network buses 1901, 1902, . . . , 1908 can also be USB”),
Although Yang substantially teaches the claimed invention, Yang fails to explicitly teach
the limitation wherein the image data further comprises a second image data;
the second image data from the first processing element to the second processing element.
In the same field, analogous art Shoaib teaches wherein the image data further comprises a second image data (see, e.g., Shoaib paragraph [0032]: “A processor module 219 includes a central processing unit (CPU) 220 and/or a graphics processing unit (GPU) 222 configured to retrieve or pull down one or more aligned images 230 from the aligned frame bus 226 and combine or composite the images”);
the second image data from the first processing element to the second processing element (see, e.g., Shoaib paragraph [0047]: “In some examples, vector data may be processed in two stages utilizing two-dimensional-processing elements in a systolic array alongside an array of one-dimensional-processing elements. For example, the G-Block 302 may process images utilizing this two stage approach. The processing elements of the array iteratively process data, passing the results of any computations to the nearest neighbors of each processing element” [i.e., second image data is transmitted to neighboring processing elements]).
Yang and Shoaib are analogous art because they are both directed to image processing and classification (see, e.g., Yang, paragraph [0079], Shoaib paragraph [0070]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang to incorporate the teachings of Shoaib to utilize second image data in the processing of image data between multiple processing elements. Doing so would have allowed Yang to use Shoaib's method for “fine-grained parallel implementations... used within various processing elements of the accelerator… The disclosed system employs arrays of specialized processing elements that are interconnected to exploit this computation pattern”, as suggested by Shoaib (see, e.g., Shoaib, paragraph [0018]).
Regarding Claim 4, as discussed above, Yang in view of Shoaib teaches the method of claim 1.
Yang further teaches wherein the preselected data comprises a tensor used by at least one of the first processing element or the second processing element to perform a CNN function (see, e.g., Yang paragraph [0055]: “To perform 3×3 convolutions at each sampling location, an example data arrangement is shown in FIG. 6. Imagery data (i.e., In(3×3)) and filter coefficients (i.e., weight coefficients C(3×3) and an offset coefficient b) are fed into an example CNN 3×3 circuitry 600” [i.e., the weights and coefficients are fed to a CNN circuitry in tensor format], paragraph [0079]: “The initial convolutional neural networks model may be obtained from many different frameworks including, but not limited to, Mxnet, caffe, tensorflow, etc.” [i.e., the frameworks used to perform convolutions of preselected data utilize tensor data structures] and paragraph [0087]: “more than one CNN based ICs are connected in series with each to store filter coefficients of a portion of the deep learning model. The large deep learning model can thus be processed with a number of CNN based ICs connected in series. An input buffer is configured in a CNN based IC for storing output data from previous CNN based IC when serially-connected” [i.e., multiple ICs (processing elements) are used in conjunction for CNN operations]).
Regarding Claim 8, as discussed above, Yang in view of Shoaib teaches the method of claim 1.
Although Yang substantially teaches the claimed invention, Yang fails to explicitly teach the limitation wherein the image sensor comprises a camera;
and wherein the image data comprises video data comprising multiple frames.
In the same field, analogous art Shoaib teaches wherein the image sensor comprises a camera (see, e.g., Shoaib paragraph [0028]: “A sensor module 201 includes a sensor 202 and a camera serial interface (CSI) 204 and/or a video interface (VI) 206 coupled to the sensor 202. In some examples, the sensor 202 is configured to capture one or more raw images 228 or frames of video, which are transmitted through the CSI 204 and/or VI 206 and transmitted to or placed onto a first frame bus (e.g., frame bus) 224” and paragraph [0049]: “Additionally or alternatively, the data sets U 406 and/or V 402 are input from an attached device such as a camera or sensor 202”);
and wherein the image data comprises video data comprising multiple frames (see, e.g., Shoaib paragraph [0028]: “the sensor 202 is configured to capture one or more raw images 228 or frames of video, which are transmitted through the CSI 204 and/or VI 206 and transmitted to or placed onto a first frame bus (e.g., frame bus) 224”).
Yang and Shoaib are analogous art because they are both directed to image processing and classification (see, e.g., Yang, paragraph [0079], Shoaib paragraph [0070]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang to incorporate the teachings of Shoaib to utilize a camera image sensor for capturing image data comprising video data of multiple frames. Doing so would have allowed Yang to use Shoaib's method in order to “use software, firmware, hardware, or a combination thereof to process a plurality of frames”, as suggested by Shoaib (see, e.g., Shoaib, paragraph [0028]).
Regarding Claim 9, as discussed above, Yang in view of Shoaib teaches the method of claim 1.
Yang further teaches wherein each of the first processing element and the second processing element are configured to perform at least one function of a preselected CNN (see, e.g., Yang paragraph [0091]: “FIG. 21 is a schematic diagram showing an example deep learning image processing system 2100 with four CNN based ICs 2101-2104 connected in series via network bus 2150. Four CNN based ICs 2101-2104 are configured to process respective portions 2011-2014 such that a larger deep learning model can be processed”),
the at least one function comprising at least one of convolution, batch normalization, pooling, or activation (see, e.g., Yang paragraph [0090]: “deep learning model 2000 divided into multiple consecutive portions is shown in FIG. 20. Deep learning model 2000 contains a number of ordered convolutional layers organized by several major convolutional layer groups separated by multiple pooling layers”).
Regarding Claim 10, as discussed above, Yang in view of Shoaib teaches the method of claim 1.
Yang further teaches wherein the first processing element and the second processing element are implemented in separate integrated circuits (see, e.g., Yang paragraph [0088]: “Referring now to FIG. 19, it is shown an example deep learning image processing system 1900. Deep learning image processing system 1900 contains multiple groups (e.g., group-1 1910, group-2 1920, . . . , group-n 1980) of CNN based ICs. Groups are connected in parallel via a network bus 1995. Each of the groups contains multiple CNN based ICs. For example, as shown in FIG. 19, group-1 1910 contains first and second CNN based ICs 1911-1912 connected in series via a network bus 1901”)
and configured to work together to implement a preselected CNN (see, e.g., Yang paragraph [0089]: “Input data that is too large to be processed by a single CNN based IC is partitioned into subsections. Subsections are then processed by respective groups of CNN based ICs. Large deep learning model is divided into portions to be handled by respective CNN based ICs connected in series within a group”).
Regarding Independent Claim 11, Yang teaches A processor dedicated to implementing a convolution neural network (CNN), comprising: (see, e.g., Yang paragraph [0036]: “The integrated circuit 100 is implemented as a digital semi-conductor chip and contains a CNN processing engine controller 110, and one or more neural networks (CNN) processing engines 102 operatively coupled to at least one input/output (I/O) data bus 120. Controller 110 is configured to control various operations of the CNN processing engines 102 for extracting features out of an input image based on an image processing technique by performing multiple layers of 3×3 convolutions with rectifications or other nonlinear operations”).
and a second processing element coupled to the first processing element via a serial communication link (see, e.g., Yang paragraph [0088]: “input buffer of a CNN based IC is configured for storing the output from a previous CNN based IC in each group of serially-connected CNN based ICs. In order to facilitate connections in series and in parallel, additional circuitry is added to the CNN based integrated circuit 100. One example bus for network bus 1995 is Universal Serial Bus (USB). Similarly, network buses 1901, 1902, . . . , 1908 can also be USB” [i.e., serial connection between CNN based ICs (processing elements) of the processing system 1900]);
wherein the first processing element3 is configured to: generate preselected data to be communicated using the serial communication link (see, e.g., Yang paragraph [0085]: “The entire set of trained coefficients or weights are pre-configured to the CNN based integrated circuit as a feature extractor for a particular data format (e.g., imagery data, voice spectrum, fingerprint, palm-print, optical character recognition (OCR), etc.). In general, there are many convolutional layers with many filters in each layer” [i.e., generation of preconfigured coefficients/weight data during training of a CNN is functionally equivalent to preselected data] and paragraph [0087]: “more than one CNN based ICs are connected in series with each to store filter coefficients of a portion of the deep learning model… An input buffer is configured in a CNN based IC for storing output data from previous CNN based IC when serially-connected”);
the image data comprising a first image data comprising multiple rows of data (see, e.g., Yang paragraph [0070]: “Imagery data received from the I/O data bus are in form of M×M pixels of imagery data in consecutive blocks… The top and the bottom rows and four corners of the received M×M pixels of imagery data are stored into respective buffers of corresponding blocks based on the geometry of the input image”);
send, via the serial communication link, a portion of the preselected data to the second processing element (see, e.g., Yang paragraph [0087]: “CNN based ICs are connected in series with each to store filter coefficients of a portion of the deep learning model. The large deep learning model can thus be processed with a number of CNN based ICs connected in series” [i.e., the coefficients (preselected data) of the neural network are distributed across a series of ICs (processing elements)] and paragraph [0088]: “Referring now to FIG. 19, it is shown an example deep learning image processing system 1900. Deep learning image processing system 1900 contains multiple groups (e.g., group-1 1910, group-2 1920, . . . , group-n 1980) of CNN based ICs. Groups are connected in parallel via a network bus 1995… input buffer of a CNN based IC is configured for storing the output from a previous CNN based IC in each group of serially-connected CNN based ICs”);
and send, via the serial communication link, a second row of the first image data to the second processing element (see, e.g., Yang paragraph [0088]: “Referring now to FIG. 19, it is shown an example deep learning image processing system 1900. Deep learning image processing system 1900 contains multiple groups (e.g., group-1 1910, group-2 1920, . . . , group-n 1980) of CNN based ICs. Groups are connected in parallel via a network bus 1995… input buffer of a CNN based IC is configured for storing the output from a previous CNN based IC in each group of serially-connected CNN based ICs” and paragraph [0092]: “An input data is partitioned into smaller subsections such that each CNN based IC can handle. In this example, the input data 2200 is partitioned… each subsection 2211 includes at least one overlapped row and column at it border with other neighboring subsection. In one embodiment, each overlapped column and row contains one-pixel thickness… each overlapped column and row contains one-pixel thickness. Four subsections 2201-2204 are processed by an example deep learning image processing system 2300 shown in FIG. 23” [i.e., multiple rows of pixel data (image input data) are sent through ICs connected in series for CNN operations]).
Although Yang substantially teaches the claimed invention, Yang fails to explicitly teach
the limitation a first processing element coupled to an image sensor;
receive image data from the image sensor,
In the same field, analogous art Shoaib teaches a first processing element coupled to an image sensor (see, e.g. Shoaib paragraph [0028]: “the sensor 202 is configured to capture one or more raw images 228 or frames of video, which are transmitted through the CSI 204 and/or VI 206 and transmitted to or placed onto a first frame bus (e.g., frame bus)” and paragraph [0029]: “An image signal processor (ISP) 208 is configured to retrieve or pull down one or more raw images 228 from the first frame bus 224 and clean up or otherwise process the raw images 228” [i.e., the ISP (processing element) is connected to the CSI image sensor via the frame bus (see, e.g., Shoaib Fig. 2)]);
receive image data from the image sensor (see, e.g., Shoaib paragraph [0028]: “A sensor module 201 includes a sensor 202 and a camera serial interface (CSI) 204 and/or a video interface (VI) 206 coupled to the sensor 202. In some examples, the sensor 202 is configured to capture one or more raw images 228 or frames of video, which are transmitted through the CSI 204 and/or VI 206 and transmitted to or placed onto a first frame bus (e.g., frame bus) 224”).
Yang and Shoaib are analogous art because they are both directed to image processing and classification (see, e.g., Yang, paragraph [0079] and Shoaib, paragraph [0070]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang to incorporate the teachings of Shoaib to utilize an image sensor coupled to a first processing element for receiving image data. Doing so would have allowed Yang to use Shoaib’s method in order to “generate a processed image”, as suggested by Shoaib (see, e.g., Shoaib, paragraph [0017]).
Regarding Claim 12, as discussed above, Yang in view of Shoaib teaches the processor of claim 11.
Yang further teaches wherein the first processing element is further configured to: (1) send, via the serial communication link, another portion of the preselected data to the second processing element (see, e.g., Yang paragraph [0087]: “CNN based ICs are connected in series with each to store filter coefficients of a portion of the deep learning model. The large deep learning model can thus be processed with a number of CNN based ICs connected in series” [i.e., the coefficients (preselected data) of the neural network are distributed across a series of ICs (processing elements)]);
(2) send, via the serial communication link, another row of the first image data to the second processing element (see, e.g., Yang paragraph [0092]: “In order to ensure proper processing between subsections, each subsection 2211 includes at least one overlapped row and column at it border with other neighboring subsection. In one embodiment, each overlapped column and row contains one-pixel thickness. Four subsections 2201-2204 are processed by an example deep learning image processing system 2300 shown in FIG. 23” [i.e., overlapped row and columns at each subsection suggests a subsequent processing of image data via the serial connection of ICs (processing elements)]);
and repeat (1) and (2) until the entire first image data has been sent or the entire preselected data has been sent (see, e.g., Yang paragraph [0077]: “The convolution-to-pooling procedures are repeated for several layers and finally connected to a Fully-connected Networks (FCN) 1560” and paragraph [0091]: “Four CNN based ICs 2101-2104 are configured to process respective portions 2011-2014 such that a larger deep learning model can be processed. In other words, a large deep leaning model is processed sequentially by a cascade of CNN based ICs connected in series” [i.e., the processing of the larger deep learning model in a sequential cascade of CNN units implies a continuous process that iteratively produces the entire image]).
Regarding Claim 13, as discussed above, Yang in view of Shoaib teaches the method of claim 12.
Yang further teaches wherein (1) and (2) were repeated until the entire first image data was sent (see, e.g., Yang paragraph [0077]: “The convolution-to-pooling procedures are repeated for several layers and finally connected to a Fully-connected Networks (FCN) 1560” and paragraph [0091]: “Four CNN based ICs 2101-2104 are configured to process respective portions 2011-2014 such that a larger deep learning model can be processed. In other words, a large deep leaning model is processed sequentially by a cascade of CNN based ICs connected in series” [i.e., the processing of the larger deep learning model in a sequential cascade of CNN units implies a continuous process that iteratively produces the entire image]);
and wherein the first processing element is further configured to: send, via the serial communication link, another portion of the preselected data to the second processing element (see, e.g., Yang paragraph [0088]: “Referring now to FIG. 19, it is shown an example deep learning image processing system 1900. Deep learning image processing system 1900 contains multiple groups (e.g., group-1 1910, group-2 1920, . . . , group-n 1980) of CNN based ICs. Groups are connected in parallel via a network bus 1995… input buffer of a CNN based IC is configured for storing the output from a previous CNN based IC in each group of serially-connected CNN based ICs” and paragraph [0092]: “the input data 2200 is partitioned into “Subsection-1” 2201, “Subsection-2” 2202, “Subsection-3” 2203 and “Subsection-4” 2204. In order to ensure proper processing between subsections, each subsection 2211 includes at least one overlapped row and column at it border with other neighboring subsection… Deep learning image processing system 2300 contains four CNN based ICs 2301-2304 connected in parallel via network bus 2350”);
and send, via the serial communication link (see, e.g., Yang paragraph [0088]: “Similarly, input buffer of a CNN based IC is configured for storing the output from a previous CNN based IC in each group of serially-connected CNN based ICs. In order to facilitate connections in series and in parallel, additional circuitry is added to the CNN based integrated circuit 100. One example bus for network bus 1995 is Universal Serial Bus (USB). Similarly, network buses 1901, 1902, . . . , 1908 can also be USB”),
Although Yang substantially teaches the claimed invention, Yang fails to explicitly teach
the limitation wherein the image data further comprises a second image data;
the second image data to the second processing element.
In the same field, analogous art Shoaib teaches wherein the image data further comprises a second image data (see, e.g., Shoaib paragraph [0032]: “A processor module 219 includes a central processing unit (CPU) 220 and/or a graphics processing unit (GPU) 222 configured to retrieve or pull down one or more aligned images 230 from the aligned frame bus 226 and combine or composite the images”);
the second image data to the second processing element (see, e.g., Shoaib paragraph [0047]: “In some examples, vector data may be processed in two stages utilizing two-dimensional-processing elements in a systolic array alongside an array of one-dimensional-processing elements. For example, the G-Block 302 may process images utilizing this two stage approach. The processing elements of the array iteratively process data, passing the results of any computations to the nearest neighbors of each processing element” [i.e., second image data is transmitted to neighboring processing elements]).
Yang and Shoaib are analogous art because they are both directed to image processing and classification (see, e.g., Yang, paragraph [0079], Shoaib paragraph [0070]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang to incorporate the teachings of Shoaib to utilize second image data in the processing of image data between multiple processing elements. Doing so would have allowed Yang to use Shoaib's method for “fine-grained parallel implementations... used within various processing elements of the accelerator… The disclosed system employs arrays of specialized processing elements that are interconnected to exploit this computation pattern”, as suggested by Shoaib (see, e.g., Shoaib, paragraph [0018]).
Regarding Claim 14, as discussed above, Yang in view of Shoaib teaches the processor of claim 11.
Yang further teaches wherein the preselected data comprises a tensor used by at least one of the first processing element or the second processing element to perform a CNN function (see, e.g., Yang paragraph [0055]: “To perform 3×3 convolutions at each sampling location, an example data arrangement is shown in FIG. 6. Imagery data (i.e., In(3×3)) and filter coefficients (i.e., weight coefficients C(3×3) and an offset coefficient b) are fed into an example CNN 3×3 circuitry 600” [i.e., the weights and coefficients are fed to a CNN circuitry in tensor format], paragraph [0079]: “The initial convolutional neural networks model may be obtained from many different frameworks including, but not limited to, Mxnet, caffe, tensorflow, etc.” [i.e., the frameworks used to perform convolutions of preselected data utilize tensor data structures] and paragraph [0087]: “more than one CNN based ICs are connected in series with each to store filter coefficients of a portion of the deep learning model. The large deep learning model can thus be processed with a number of CNN based ICs connected in series. An input buffer is configured in a CNN based IC for storing output data from previous CNN based IC when serially-connected” [i.e., multiple ICs (processing elements) are used in conjunction for CNN operations]).
Regarding Claim 18, as discussed above, Yang in view of Shoaib teaches the processor of claim 11.
Although Yang substantially teaches the claimed invention, Yang fails to explicitly teach the limitation wherein the image sensor comprises a camera;
and wherein the image data comprises video data comprising multiple frames.
In the same field, analogous art Shoaib teaches wherein the image sensor comprises a camera (see, e.g., Shoaib paragraph [0028]: “A sensor module 201 includes a sensor 202 and a camera serial interface (CSI) 204 and/or a video interface (VI) 206 coupled to the sensor 202. In some examples, the sensor 202 is configured to capture one or more raw images 228 or frames of video, which are transmitted through the CSI 204 and/or VI 206 and transmitted to or placed onto a first frame bus (e.g., frame bus) 224” and paragraph [0049]: “Additionally or alternatively, the data sets U 406 and/or V 402 are input from an attached device such as a camera or sensor 202”);
and wherein the image data comprises video data comprising multiple frames (see, e.g., Shoaib paragraph [0028]: “the sensor 202 is configured to capture one or more raw images 228 or frames of video, which are transmitted through the CSI 204 and/or VI 206 and transmitted to or placed onto a first frame bus (e.g., frame bus) 224”).
Yang and Shoaib are analogous art because they are both directed to image processing and classification (see, e.g., Yang, paragraph [0079], Shoaib paragraph [0070]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang to incorporate the teachings of Shoaib to utilize a camera image sensor for capturing image data comprising video data of multiple frames. Doing so would have allowed Yang to use Shoaib's method in order to “use software, firmware, hardware, or a combination thereof to process a plurality of frames”, as suggested by Shoaib (see, e.g., Shoaib, paragraph [0028]).
Regarding Claim 19, as discussed above, Yang in view of Shoaib teaches the processor of claim 11.
Yang further teaches wherein each of the first processing element and the second processing element are configured to perform at least one function of a preselected CNN (see, e.g., Yang paragraph [0091]: “FIG. 21 is a schematic diagram showing an example deep learning image processing system 2100 with four CNN based ICs 2101-2104 connected in series via network bus 2150. Four CNN based ICs 2101-2104 are configured to process respective portions 2011-2014 such that a larger deep learning model can be processed”),
the at least one function comprising at least one of convolution, batch normalization, pooling, or activation (see, e.g., Yang paragraph [0090]: “deep learning model 2000 divided into multiple consecutive portions is shown in FIG. 20. Deep learning model 2000 contains a number of ordered convolutional layers organized by several major convolutional layer groups separated by multiple pooling layers”).
Regarding Claim 20, as discussed above, Yang in view of Shoaib teaches the processor of claim 11.
Yang further teaches wherein the first processing element and the second processing element are implemented in separate integrated circuits (see, e.g., Yang paragraph [0088]: “Referring now to FIG. 19, it is shown an example deep learning image processing system 1900. Deep learning image processing system 1900 contains multiple groups (e.g., group-1 1910, group-2 1920, . . . , group-n 1980) of CNN based ICs. Groups are connected in parallel via a network bus 1995. Each of the groups contains multiple CNN based ICs. For example, as shown in FIG. 19, group-1 1910 contains first and second CNN based ICs 1911-1912 connected in series via a network bus 1901”)
and configured to work together to implement a preselected CNN (see, e.g., Yang paragraph [0089]: “Input data that is too large to be processed by a single CNN based IC is partitioned into subsections. Subsections are then processed by respective groups of CNN based ICs. Large deep learning model is divided into portions to be handled by respective CNN based ICs connected in series within a group”).
14. Claims 5, 7, 15 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Shoaib and further in view of Krig (US 20140181472 A1; hereinafter Krig).
Regarding Claim 5, as discussed above, Yang in view of Shoaib teaches the method of claim 1.
Although Yang in view of Shoaib substantially teaches the claimed invention, Yang in view of Shoaib fails to explicitly teach the limitation wherein the serial communication link is a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI).
In the same field, analogous art Krig teaches wherein the serial communication link is a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI) (see, e.g., Krig paragraph [0023]: “The computing device 100 includes an image capture mechanism 108… the image capture mechanism may be a camera device that interfaces with the scalable compute fabric 102 using an interface developed according to specifications by the Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) Alliance, For example, the camera serial interface may be a MIPI CSI-1 Interface”).
Yang, Shoaib and Krig are analogous art because they are each directed to image data processing (see, e.g., Yang, paragraph [0036], Shoaib paragraph [0054], and Krig paragraph [0039]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang in view of Shoaib to incorporate the teachings of Krig to utilize a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI) as a serial communication link between Processing elements. Doing so would have allowed Yang in view of Shoaib to use Krig's method in order to support a “scalable compute fabric in which the compute elements are available for use as needed, and dynamically configurable for assignments to special purpose pipelines across I/O connections and busses between compute elements”, as suggested by Krig (see, e.g., Krig, paragraph [0010]).
Regarding Claim 7, as discussed above, as discussed above, Yang in view of Shoaib teaches the method of claim 1.
Although Yang in view of Shoaib substantially teaches the claimed invention, Yang in view of Shoaib fails to explicitly teach the limitation wherein the serial communication link is implemented using at least one of a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI), a High-Definition Multimedia Interface (HDMI), or a Serializer/Deserializer (SerDes).
In the same field, analogous art Krig teaches wherein the serial communication link is implemented using at least one of a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI), a High-Definition Multimedia Interface (HDMI), or a Serializer/Deserializer (SerDes) (see, e.g., Krig paragraph [0023]: “The computing device 100 includes an image capture mechanism 108… the image capture mechanism may be a camera device that interfaces with the scalable compute fabric 102 using an interface developed according to specifications by the Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) Alliance, For example, the camera serial interface may be a MIPI CSI-1 Interface”).
Yang, Shoaib and Krig are analogous art because they are each directed to image data processing (see, e.g., Yang, paragraph [0036], Shoaib paragraph [0054], and Krig paragraph [0039]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang in view of Shoaib to incorporate the teachings of Krig to utilize a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI), a High-Definition Multimedia Interface (HDMI), or a Serializer/Deserializer (SerDes) as a serial communication link between Processing elements. Doing so would have allowed Yang in view of Shoaib to use Krig's method in order to support a “scalable compute fabric in which the compute elements are available for use as needed, and dynamically configurable for assignments to special purpose pipelines across I/O connections and busses between compute elements”, as suggested by Krig (see, e.g., Krig, paragraph [0010]).
Regarding Claim 15, as discussed above, Yang in view of Shoaib teaches the processor of claim 11.
Although Yang in view of Shoaib substantially teaches the claimed invention, Yang in view of Shoaib fails to explicitly teach the limitation wherein the serial communication link is a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI).
In the same field, analogous art Krig teaches wherein the serial communication link is a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI) (see, e.g., Krig paragraph [0023]: “The computing device 100 includes an image capture mechanism 108… the image capture mechanism may be a camera device that interfaces with the scalable compute fabric 102 using an interface developed according to specifications by the Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) Alliance, For example, the camera serial interface may be a MIPI CSI-1 Interface”).
Yang, Shoaib and Krig are analogous art because they are each directed to image data processing (see, e.g., Yang, paragraph [0036], Shoaib paragraph [0054], and Krig paragraph [0039]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang in view of Shoaib to incorporate the teachings of Krig to utilize a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI) as a serial communication link between Processing elements. Doing so would have allowed Yang in view of Shoaib to use Krig's method in order to support a “scalable compute fabric in which the compute elements are available for use as needed, and dynamically configurable for assignments to special purpose pipelines across I/O connections and busses between compute elements”, as suggested by Krig (see, e.g., Krig, paragraph [0010]).
Regarding Claim 17, as discussed above, as discussed above, Yang in view of Shoaib teaches the processor of claim 11.
Although Yang in view of Shoaib substantially teaches the claimed invention, Yang in view of Shoaib fails to explicitly teach the limitation wherein the serial communication link is implemented using at least one of a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI), a High-Definition Multimedia Interface (HDMI), or a Serializer/Deserializer (SerDes).
In the same field, analogous art Krig teaches wherein the serial communication link is implemented using at least one of a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI), a High-Definition Multimedia Interface (HDMI), or a Serializer/Deserializer (SerDes) (see, e.g., Krig paragraph [0023]: “The computing device 100 includes an image capture mechanism 108… the image capture mechanism may be a camera device that interfaces with the scalable compute fabric 102 using an interface developed according to specifications by the Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) Alliance, For example, the camera serial interface may be a MIPI CSI-1 Interface”).
Yang, Shoaib and Krig are analogous art because they are each directed to image data processing (see, e.g., Yang, paragraph [0036], Shoaib paragraph [0054], and Krig paragraph [0039]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang in view of Shoaib to incorporate the teachings of Krig to utilize a Camera Serial Interface (CSI) of the Mobile Industry Processor Interface (MIPI), a High-Definition Multimedia Interface (HDMI), or a Serializer/Deserializer (SerDes) as a serial communication link between Processing elements. Doing so would have allowed Yang in view of Shoaib to use Krig's method in order to support a “scalable compute fabric in which the compute elements are available for use as needed, and dynamically configurable for assignments to special purpose pipelines across I/O connections and busses between compute elements”, as suggested by Krig (see, e.g., Krig, paragraph [0010]).
15. Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Yang and Shoaib in view of Krig and further in view of Iverson (US 20210210046 A1; hereinafter Iverson). Iverson was filed on 5/23/2019, and this date is before the earliest effective filing
date of this application, i.e., 7/19/2019. Therefore, Iverson constitutes prior art under 35 U.S.C.
102(a)(2).
Regarding Claim 6, as discussed above, Krig in view of Shoaib and further in view of Krig teaches the method of claim 5.
Yang further teaches wherein the sending, via the serial communication link (see, e.g., Yang paragraph [0088]: “Referring now to FIG. 19, it is shown an example deep learning image processing system 1900. Deep learning image processing system 1900 contains multiple groups (e.g., group-1 1910, group-2 1920, . . . , group-n 1980) of CNN based ICs. Groups are connected in parallel via a network bus 1995… input buffer of a CNN based IC is configured for storing the output from a previous CNN based IC in each group of serially-connected CNN based ICs”),
Although Yang in view of Shoaib and further in view of Krig substantially teaches the claimed invention, Yang in view of Shoaib and further in view of Krig fails to explicitly teach the first row of the first image data from the first processing element to the second processing element comprises sending the first row of the first image data in a payload portion of a MIPI packet,
wherein the MIPI packet comprises the payload portion and a metadata portion;
and wherein the sending, via the serial communication link, the portion of the preselected data comprises sending the portion of the preselected data in the metadata portion of the MIPI packet.
In the same field, analogous art Iverson teaches the first row of the first image data from the first processing element to the second processing element comprises sending the first row of the first image data in a payload portion of a MIPI packet (see, e.g., Iverson paragraph [0051]: “the drive scheme module 122 formats the image data frames 140 into MIPI image frame”, paragraph [0057]: “Examples of operations may include, but are not limited to, receive the data frames 140, search the data frames 140 for one or more synchronization bytes that identify a portion (e.g., the first row) of a data frame, and map portions (e.g., bytes, rows, columns, etc.) of the data frames 140 to predetermined variables”, paragraph [0062]: “The one or more processors 154 are configured to read and execute the sensor data acquisition module and the image data processing module 104 from a first memory 156” and paragraph [0090]: “The MIPI ‘packet’ structure may be based on 24-bits of data. 24 bits may be used to represent a single “pixel” in what would can be considered as the ‘active-image portion’ of the MIPI data-stream”),
wherein the MIPI packet comprises the payload portion and a metadata portion (see, e.g., Iverson paragraph [0051]: “According to one embodiment of the disclosure, the drive scheme module 122 formats the image data frames 140 into MIPI image frame. The MIPI image frames are modified to replace some of the pixels of an image frame with drive scheme (control structure) information” [i.e., the drive scheme/control structure information of the MIPI image frame constitutes a metadata portion], and paragraph [0090]: “The MIPI ‘packet’ structure may be based on 24-bits of data. 24 bits may be used to represent a single ‘pixel’ in what would can be considered as the ‘active-image portion’ of the MIPI data-stream” [i.e., the active-image portion of the MIPI data-stream functions as a payload portion of the MIPI packet]);
and wherein the sending, via the serial communication link, the portion of the preselected data comprises sending the portion of the preselected data in the metadata portion of the MIPI packet (see, e.g., Iverson paragraph [0026-0027]: “The image data frames 140 may be formatted in accordance with, for example, one or more MIPI (“mobile industry processor interface”) or modified-MIPI interfaces or communication protocols. The drive scheme module 122 may be configured to transmit the image data frames 140 to the display driver module 106 over a communications channel 142 (e.g., a conductive bus, a network, a wireless interface, etc.)… The display driver module 106 may be configured to operate the display 108 using the image data frames 140 received from the drive scheme module 122. The display driver module 140 may use information (e.g., the drive scheme 134) contained within the image data frames 140 to operate the display 108” [i.e., image data frames function as preselected data] and paragraph [0053]: “In an embodiment, the drive scheme module 122 inserts the drive scheme 134 into the blanking intervals of a MIPI protocol data frames to transport the drive scheme 134, with the image frame 128 to the display driver 106” [i.e., the drive scheme is inserted into an MIPI packet data frames (MIPI packet)]).
Yang, Shoaib, Krig and Iverson are analogous art because they are each directed to image data processing (see, e.g., Yang, paragraph [0036], Shoaib paragraph [0054], Krig paragraph [0039], and Iverson paragraph [0038]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang in view of Shoaib and further in view of Krig to incorporate the teachings of Iverson to send a row of first image data in a payload portion of a MIPI packet from one processing element to another, where the MIPI packet comprises a payload portion and a metadata portion, and sending a portion of preselected data in the metadata portion of the MIPI packet via a serial communication link. Doing so would have allowed Yang in view of Shoaib and further in view of Krig to use Iverson's method in order to “enable[s] the recipient of the data frames to extract information from the data frames”, as suggested by Iverson (see, e.g., Iverson, paragraph [0050]).
Regarding Claim 16, as discussed above, Krig in view of Shoaib and further in view of Krig teaches the processor of claim 15.
Yang further teaches wherein the first processing element is configured to send, via the serial communication link (see, e.g., Yang paragraph [0088]: “Referring now to FIG. 19, it is shown an example deep learning image processing system 1900. Deep learning image processing system 1900 contains multiple groups (e.g., group-1 1910, group-2 1920, . . . , group-n 1980) of CNN based ICs. Groups are connected in parallel via a network bus 1995… input buffer of a CNN based IC is configured for storing the output from a previous CNN based IC in each group of serially-connected CNN based ICs”),
Although Yang in view of Shoaib and further in view of Krig substantially teaches the claimed invention, Yang in view of Shoaib and further in view of Krig fails to explicitly teach a first row of the first image data to the second processing element by being further configured to send the first row of the first image data in a payload portion of a MIPI packet, wherein the MIPI packet comprises the payload portion and a metadata portion;
and wherein the first processing element is configured to send, via the serial communication link, the portion of the preselected data by being further configured to send the portion of the preselected data in the metadata portion of the MIPI packet.
In the same field, analogous art Iverson teaches a first row of the first image data to the second processing element by being further configured to send the first row of the first image data in a payload portion of a MIPI packet (see, e.g., Iverson paragraph [0051]: “the drive scheme module 122 formats the image data frames 140 into MIPI image frame”, paragraph [0057]: “Examples of operations may include, but are not limited to, receive the data frames 140, search the data frames 140 for one or more synchronization bytes that identify a portion (e.g., the first row) of a data frame, and map portions (e.g., bytes, rows, columns, etc.) of the data frames 140 to predetermined variables”, paragraph [0062]: “The one or more processors 154 are configured to read and execute the sensor data acquisition module and the image data processing module 104 from a first memory 156” and paragraph [0090]: “The MIPI ‘packet’ structure may be based on 24-bits of data. 24 bits may be used to represent a single “pixel” in what would can be considered as the ‘active-image portion’ of the MIPI data-stream”),
wherein the MIPI packet comprises the payload portion and a metadata portion (see, e.g., Iverson paragraph [0051]: “According to one embodiment of the disclosure, the drive scheme module 122 formats the image data frames 140 into MIPI image frame. The MIPI image frames are modified to replace some of the pixels of an image frame with drive scheme (control structure) information” [i.e., the drive scheme/control structure information of the MIPI image frame constitutes a metadata portion], and paragraph [0090]: “The MIPI ‘packet’ structure may be based on 24-bits of data. 24 bits may be used to represent a single ‘pixel’ in what would can be considered as the ‘active-image portion’ of the MIPI data-stream” [i.e., the active-image portion of the MIPI data-stream functions as a payload portion of the MIPI packet]);
and wherein the first processing element is configured to send, via the serial communication link, the portion of the preselected data by being further configured to send the portion of the preselected data in the metadata portion of the MIPI packet (see, e.g., Iverson paragraph [0026-0027]: “The image data frames 140 may be formatted in accordance with, for example, one or more MIPI (“mobile industry processor interface”) or modified-MIPI interfaces or communication protocols. The drive scheme module 122 may be configured to transmit the image data frames 140 to the display driver module 106 over a communications channel 142 (e.g., a conductive bus, a network, a wireless interface, etc.)… The display driver module 106 may be configured to operate the display 108 using the image data frames 140 received from the drive scheme module 122. The display driver module 140 may use information (e.g., the drive scheme 134) contained within the image data frames 140 to operate the display 108” [i.e., image data frames function as preselected data] and paragraph [0053]: “In an embodiment, the drive scheme module 122 inserts the drive scheme 134 into the blanking intervals of a MIPI protocol data frames to transport the drive scheme 134, with the image frame 128 to the display driver 106” [i.e., the drive scheme is inserted into an MIPI packet data frames (MIPI packet)]).
Yang, Shoaib, Krig and Iverson are analogous art because they are each directed to image data processing (see, e.g., Yang, paragraph [0036], Shoaib paragraph [0054], Krig paragraph [0039], and Iverson paragraph [0038]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang in view of Shoaib and further in view of Krig to incorporate the teachings of Iverson to send a row of first image data in a payload portion of a MIPI packet from one processing element to another, where the MIPI packet comprises a payload portion and a metadata portion, and sending a portion of preselected data in the metadata portion of the MIPI packet via a serial communication link. Doing so would have allowed Yang in view of Shoaib and further in view of Krig to use Iverson's method in order to “enable[s] the recipient of the data frames to extract information from the data frames”, as suggested by Iverson (see, e.g., Iverson, paragraph [0050]).
Conclusion
16. Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEREMY A HALZEL whose telephone number is (571)272-5290. The examiner can normally be reached Mon-Fri 7:30-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEREMY HALZEL/Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
1 As discussed above in the section 112(f) interpretation of this claim, the term “processing element” has been
interpreted as the specified combination of software and/or hardware disclosed in the specification for this
specific component that is capable of performing the claimed functions
2 As discussed above in the section 112(f) interpretation of this claim, the term “processing elements” has been
interpreted as the specified combination of software and/or hardware disclosed in the specification for this
specific component that is capable of performing the claimed functions
3 As discussed above in the section 112(f) interpretation of this claim, the term “processing element” has been
interpreted as the specified combination of software and/or hardware disclosed in the specification for this
specific component that is capable of performing the claimed functions
Prosecution Insights