DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, or 365(c) is acknowledged. In addition, acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. 18/860,726, filed on October 28, 2024.
Oath/Declaration
Oath/Declaration as filed on October 28, 2024 is noted by the Examiner.
Claim Objections
Claim 3 is objected to because of the following informalities:
In particular, limitation term “a CV process” in second line of the renders the claim indefinite, because the meaning of the acronym is not apparent. Examiner recommends applicant amend the claim 3, without adding new matter, in a way that clarifies the acronym. See MPEP § 2173.05(a). Accordingly, any claim(s) dependent on claim 3 are objected to based on same above reasoning.
Claim Interpretation – 35 USC § 112(f)
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may
be expressed as a means or step for performing a specified function without the recital of
structure, material, or acts in support thereof, and such claim shall be construed to cover
the corresponding structure, material, or acts described in the specification and
equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for
performing a specified function without the recital of structure, material, or acts in
support thereof, and such claim shall be construed to cover the corresponding structure,
material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) are: “an image processing unit that” and “an inference processing unit that” recited in sixth thru ninth lines of claim 1, fourth thru seventh lines of claim 8, and “fifth thru eleventh lines of claim 9” are both considered to read on AI image processing unit 44 (pgs. 52 and 59, Lines 1-8 and Lines 16-18 respectively; 44 FIG. 8).
Because these claim limitation(s) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, and 5-9 are rejected under 35 U.S.C. 102(a)(1) and 102(a)(2) as being anticipated by Schmidt et al., U.S. Patent 11,182,672 B1 (hereinafter Schmidt).
Regarding claim 1, Schmidt teaches an image sensor comprising: a pixel array unit where multiple pixels are two-dimensionally arrayed (108 and 704 FIGS. 7-8, Cols. 5 and 6, Lines 60-67 and Lines 1-15, respectively of Schmidt teach FIG. 7 is a block diagram of an imaging system 108 in accordance with embodiments of the present disclosure; the imaging system 108 generally includes a digital focal plane array 704; the imaging system 108 also includes reconfigurable focal plane electronics (RFPE) 708; in accordance with at least some embodiments of the present disclosure, the RFPE 708 implements a microsequencer 712; the imaging system 708 can also include or can be connected to a host system 716; in addition, the imaging system can include a lens or optical assembly 720 for focusing collected light onto the focal plane array 704; as an example, but without limitation, the focal plane array 704 can include a backside illuminated CMOS image sensor having a 1024×1024 two-dimensional array of pixels; and as can be appreciated by one of skill in the art after consideration of the present disclosure, in addition to a focal plane array formed from a plurality of photosensitive sites or pixels, the focal plane array 708 can incorporate or be associated with driver and analog-to-digital conversion (ADC) circuitry, enabling the focal plane array 704 to provide a digital output representative of an amplitude or intensity of light detected at each pixel within the focal plane array 704, and See also at least ABSTRACT and Cols. 6 and 7, Lines 16-67 and Lines 1-16, respectively of Schmidt (i.e., Schmidt teaches an imaging method and system with a digital focal plane array that includes CMOS image sensor having a two-dimensional array of pixels));
a frame memory that stores image data output from the pixel array unit; an image processing unit that performs image processing for the image data stored in the frame memory (724, 732 FIGS. 7-8, Col. 7, Lines 17-63 of Schmidt teaches FIG. 8 depicts aspects of a deep learning network 738 in accordance with embodiments the present disclosure; the deep learning network 738 can be implemented by the RFPE 708 through execution of application instructions 736 stored in the RFPE memory 724 and executed by the RFPE processor 732; initially, at a data acquisition or input stage (step 804), an imaging system is operated to take multiple raw images of the same or different scenes 104; during a training phase, the images may contain objects 120 in the form of targets of interest for which automatic classification and identification by the deep learning network 738, after training, is desired; moreover, during training, the images may be obtained by an imaging system that is separate from and/or different than the imaging system 108; in accordance with other embodiments of the present disclosure, the imaging system 108 or an instance of the imaging system 108 that will be operated in connection with a trained deep learning network 738 as described herein can be used to obtain some or all of the images used for training; preprocessing is then performed (step 808); in preprocessing, some or all of the multiple images obtained as part of the data acquisition step are preprocessed to remove noise and image defects; more particularly, preprocessing can include frame co-addition, nonuniformity correction (NUC), compression, and data formatting processes; during a training phase, preprocessing may be performed using processing components that are different than and/or or provided separately from the RFPE 708; a reconfigurable focal plane electronics (RFPE) digital module 708 provided as part of an imaging system 108 can also be used to preprocess the multiple raw images to remove the different noise and image defects; moreover, in a deployed imaging system 108, the RFPE digital module 708 receives the multiple images obtained by the focal plane array 704 as part of the data acquisition step, and preprocesses those images to remove noise and image defects; in accordance with embodiments of the present disclosure, the RFPE digital module 708 can comprise a plurality of processing boards, among which the image data is distributed; as an example, but without limitation, each processing board can receive an aggregate of image data at a rate of up to 50 Gbps; the extent of image processing conducted in the digital board can vary for each application; additional or alternative preprocessing steps can also be taken; and the processed image data is then stored in memory (e.g. memory 724) or data storage (e.g. data storage 728) for later deep learning with optimized selected vector concatenation (step 812), and See also at least ABSTRACT and Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 16-67 and Lines 1-16, Lines 64-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches the method and system, which has the digital focal plane array that includes the CMOS image sensor having the two-dimensional array of pixels, for obtaining and processing image data, by RFPE processor that includes one or more multi-threaded processors that perform image and graphical processing and provides images to a deep learning network using a highly parallel structure of data paths to manipulate large blocks of data, wherein instructions executable by the RFPE processor and the image data, obtained by the focal plane array, are stored in memory or data storage)); and
an inference processing unit that performs an inference process using an artificial intelligence model on a basis of, as an input tensor, the image data for which image processing has been performed by the image processing unit (732, 738 FIGS. 7-8, Col. 10, Lines 1-45 of Schmidt teaches in processing actual or live image data, the process branches from step 816 to step 836, where the image data collected by the imaging system 108 is preprocessed and then processed using the OSVC model of the trained deep learning network 738; more particularly, the model can be used to return an output (step 840) in the form of a plurality of classified objects 120 corresponding to targets of interest to a user; the output results can thus include automated object classification, object identification to enable inferencing insights about target image objects 120 in a real image, and pose information to enable inferring further insights about target objects 120 in the real image; therefore, operation of a deployed network 704 can include identifying objects 120 within an image, and then selectively propagating pose information for some identified objects 120, and not propagating pose information for other identified objects 120; the operation of the deep learning network 738 can further include selecting a relevant set of images or areas within the full frame images that contain objects 120 of interest; moreover, the imaging system 108 can operate to identify those objects 120 of interest based on vector information, such as the pose of elements of an object within the scene or the relative pose of objects 120 to one another; other objects can be identified, but not associated with vector information that is propagated through the deep learning networks 738; for example, a lamp post may be identified by the deep learning network, but position and location information associated with the lamp post is not propagated through the deep learning network 738; in accordance with further embodiments of the present disclosure, the deep learning network 738 can select some or all images or sub-images containing objects of interest 120 for further analysis (i.e. potential autonomous action) or for delivery to a human analyst or to a system separate from the imaging system 108, for example via the communications interface 748 included as part of the host system 716; therefore, embodiments of the present disclosure can improve runtime performance of the deep learning network 738 itself; and moreover, the runtime performance of autonomous applications that utilize image information selected by the deep learning network 738 of communications networks associated with a communication interface 748, and/or the workload of human analyst or other systems can be improved, and See also at least Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 16-67 and Lines 1-16, Lines 17-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches the deep learning network implemented by the reconfigurable focal plane electronics (RFPE), which includes the RFPE processor that further includes the one or more multi-threaded processors that perform image and graphical processing and provides images to the deep learning network using the highly parallel structure of the data paths to manipulate the large blocks of data, that performs the processing of images and that also includes training the deep learning network to predict a classification and identification of targets of interest and enable inferencing insights about target image objects in a real image, wherein during the training an HCN model is trained using images containing different poses of targets of interest and then optimized to form an OSVC model to subsequently automatically classify and identify objects in real image data).
Regarding claim 2, Schmidt teaches the image sensor according to claim 1, wherein the image processing unit and the inference processing unit are provided as different processors (FIGS. 7-8, Col. 10, Lines 1-45 of Schmidt teaches in processing actual or live image data, the process branches from step 816 to step 836, where the image data collected by the imaging system 108 is preprocessed and then processed using the OSVC model of the trained deep learning network 738; more particularly, the model can be used to return an output (step 840) in the form of a plurality of classified objects 120 corresponding to targets of interest to a user; the output results can thus include automated object classification, object identification to enable inferencing insights about target image objects 120 in a real image, and pose information to enable inferring further insights about target objects 120 in the real image; therefore, operation of a deployed network 704 can include identifying objects 120 within an image, and then selectively propagating pose information for some identified objects 120, and not propagating pose information for other identified objects 120; the operation of the deep learning network 738 can further include selecting a relevant set of images or areas within the full frame images that contain objects 120 of interest; moreover, the imaging system 108 can operate to identify those objects 120 of interest based on vector information, such as the pose of elements of an object within the scene or the relative pose of objects 120 to one another; other objects can be identified, but not associated with vector information that is propagated through the deep learning networks 738; for example, a lamp post may be identified by the deep learning network, but position and location information associated with the lamp post is not propagated through the deep learning network 738; in accordance with further embodiments of the present disclosure, the deep learning network 738 can select some or all images or sub-images containing objects of interest 120 for further analysis (i.e. potential autonomous action) or for delivery to a human analyst or to a system separate from the imaging system 108, for example via the communications interface 748 included as part of the host system 716; therefore, embodiments of the present disclosure can improve runtime performance of the deep learning network 738 itself; and moreover, the runtime performance of autonomous applications that utilize image information selected by the deep learning network 738 of communications networks associated with a communication interface 748, and/or the workload of human analyst or other systems can be improved, and See also at least Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 16-67 and Lines 1-16, Lines 17-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches a deep learning network implemented by the reconfigurable focal plane electronics (RFPE), which includes the RFPE processor that further includes the one or more multi-threaded processors, that perform image and graphical processing and provides images to and includes the deep learning network to predict a classification and identification of targets of interest and enable inferencing insights about target image objects in a real image, wherein during the training an HCN model is trained using images containing different poses of targets of interest and then optimized to form an OSVC model to subsequently automatically classify and identify objects in real image data).
Regarding claim 3, Schmidt teaches the image sensor according to claim 2, wherein the image processing unit performs a CV process (FIGS. 7-8, Col. 10, Lines 1-45 of Schmidt teaches in processing actual or live image data, the process branches from step 816 to step 836, where the image data collected by the imaging system 108 is preprocessed and then processed using the OSVC model of the trained deep learning network 738; more particularly, the model can be used to return an output (step 840) in the form of a plurality of classified objects 120 corresponding to targets of interest to a user; the output results can thus include automated object classification, object identification to enable inferencing insights about target image objects 120 in a real image, and pose information to enable inferring further insights about target objects 120 in the real image; therefore, operation of a deployed network 704 can include identifying objects 120 within an image, and then selectively propagating pose information for some identified objects 120, and not propagating pose information for other identified objects 120; the operation of the deep learning network 738 can further include selecting a relevant set of images or areas within the full frame images that contain objects 120 of interest; moreover, the imaging system 108 can operate to identify those objects 120 of interest based on vector information, such as the pose of elements of an object within the scene or the relative pose of objects 120 to one another; other objects can be identified, but not associated with vector information that is propagated through the deep learning networks 738; for example, a lamp post may be identified by the deep learning network, but position and location information associated with the lamp post is not propagated through the deep learning network 738; in accordance with further embodiments of the present disclosure, the deep learning network 738 can select some or all images or sub-images containing objects of interest 120 for further analysis (i.e. potential autonomous action) or for delivery to a human analyst or to a system separate from the imaging system 108, for example via the communications interface 748 included as part of the host system 716; therefore, embodiments of the present disclosure can improve runtime performance of the deep learning network 738 itself; and moreover, the runtime performance of autonomous applications that utilize image information selected by the deep learning network 738 of communications networks associated with a communication interface 748, and/or the workload of human analyst or other systems can be improved, and See also at least Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 16-67 and Lines 1-16, Lines 17-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches a deep learning network implemented by the reconfigurable focal plane electronics (RFPE), which includes the RFPE processor that further includes the one or more multi-threaded processors, that performs the processing of images that also includes training the deep learning network to predict a classification and identification of targets of interest and enable inferencing insights about target image objects in a real image, wherein during the training an HCN model is trained using images containing different poses of targets of interest and then optimized to form an OSVC model to subsequently automatically classify and identify objects in real image data).
Regarding claim 5, Schmidt teaches the image sensor according to claim 2, wherein the image processing unit generates an input tensor of the artificial intelligence model (FIGS. 7-8, Col. 10, Lines 1-45 of Schmidt teaches in processing actual or live image data, the process branches from step 816 to step 836, where the image data collected by the imaging system 108 is preprocessed and then processed using the OSVC model of the trained deep learning network 738; more particularly, the model can be used to return an output (step 840) in the form of a plurality of classified objects 120 corresponding to targets of interest to a user; the output results can thus include automated object classification, object identification to enable inferencing insights about target image objects 120 in a real image, and pose information to enable inferring further insights about target objects 120 in the real image; therefore, operation of a deployed network 704 can include identifying objects 120 within an image, and then selectively propagating pose information for some identified objects 120, and not propagating pose information for other identified objects 120; the operation of the deep learning network 738 can further include selecting a relevant set of images or areas within the full frame images that contain objects 120 of interest; moreover, the imaging system 108 can operate to identify those objects 120 of interest based on vector information, such as the pose of elements of an object within the scene or the relative pose of objects 120 to one another; other objects can be identified, but not associated with vector information that is propagated through the deep learning networks 738; for example, a lamp post may be identified by the deep learning network, but position and location information associated with the lamp post is not propagated through the deep learning network 738; in accordance with further embodiments of the present disclosure, the deep learning network 738 can select some or all images or sub-images containing objects of interest 120 for further analysis (i.e. potential autonomous action) or for delivery to a human analyst or to a system separate from the imaging system 108, for example via the communications interface 748 included as part of the host system 716; therefore, embodiments of the present disclosure can improve runtime performance of the deep learning network 738 itself; and moreover, the runtime performance of autonomous applications that utilize image information selected by the deep learning network 738 of communications networks associated with a communication interface 748, and/or the workload of human analyst or other systems can be improved, and See also at least Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 16-67 and Lines 1-16, Lines 17-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches the deep learning network implemented by the reconfigurable focal plane electronics (RFPE), which includes the RFPE processor that further includes the one or more multi-threaded processors that perform image and graphical processing and provides images to the deep learning network, that performs the processing of images that also includes training the deep learning network to predict a classification and identification of targets of interest and enable inferencing insights about target image objects in a real image, wherein during the training an HCN model is trained using images containing different poses of targets of interest and then optimized to form an OSVC model to subsequently automatically classify and identify objects in real image data).
Regarding claim 6, Schmidt teaches an image processing method executed by a computer device, the image processing method (108 FIGS. 7-8, Cols. 5 and 6, Lines 60-67 and Lines 1-15, respectively of Schmidt teach FIG. 7 is a block diagram of an imaging system 108 in accordance with embodiments of the present disclosure; the imaging system 108 generally includes a digital focal plane array 704; the imaging system 108 also includes reconfigurable focal plane electronics (RFPE) 708; in accordance with at least some embodiments of the present disclosure, the RFPE 708 implements a microsequencer 712; the imaging system 708 can also include or can be connected to a host system 716; in addition, the imaging system can include a lens or optical assembly 720 for focusing collected light onto the focal plane array 704; as an example, but without limitation, the focal plane array 704 can include a backside illuminated CMOS image sensor having a 1024×1024 two-dimensional array of pixels; and as can be appreciated by one of skill in the art after consideration of the present disclosure, in addition to a focal plane array formed from a plurality of photosensitive sites or pixels, the focal plane array 708 can incorporate or be associated with driver and analog-to-digital conversion (ADC) circuitry, enabling the focal plane array 704 to provide a digital output representative of an amplitude or intensity of light detected at each pixel within the focal plane array 704, and See also at least ABSTRACT and Cols. 6 and 7, Lines 16-67 and Lines 1-16, respectively of Schmidt (i.e., Schmidt teaches an imaging method and system with a digital focal plane array that includes CMOS image sensor having a two-dimensional array of pixels)) comprising: a process for storing image data output from a pixel array unit where multiple pixels are two-dimensionally arrayed; image processing for the stored image data (704 FIGS. 7-8, Col. 7, Lines 17-63 of Schmidt teaches FIG. 8 depicts aspects of a deep learning network 738 in accordance with embodiments the present disclosure; the deep learning network 738 can be implemented by the RFPE 708 through execution of application instructions 736 stored in the RFPE memory 724 and executed by the RFPE processor 732; initially, at a data acquisition or input stage (step 804), an imaging system is operated to take multiple raw images of the same or different scenes 104; during a training phase, the images may contain objects 120 in the form of targets of interest for which automatic classification and identification by the deep learning network 738, after training, is desired; moreover, during training, the images may be obtained by an imaging system that is separate from and/or different than the imaging system 108; in accordance with other embodiments of the present disclosure, the imaging system 108 or an instance of the imaging system 108 that will be operated in connection with a trained deep learning network 738 as described herein can be used to obtain some or all of the images used for training; preprocessing is then performed (step 808); in preprocessing, some or all of the multiple images obtained as part of the data acquisition step are preprocessed to remove noise and image defects; more particularly, preprocessing can include frame co-addition, nonuniformity correction (NUC), compression, and data formatting processes; during a training phase, preprocessing may be performed using processing components that are different than and/or or provided separately from the RFPE 708; a reconfigurable focal plane electronics (RFPE) digital module 708 provided as part of an imaging system 108 can also be used to preprocess the multiple raw images to remove the different noise and image defects; moreover, in a deployed imaging system 108, the RFPE digital module 708 receives the multiple images obtained by the focal plane array 704 as part of the data acquisition step, and preprocesses those images to remove noise and image defects; in accordance with embodiments of the present disclosure, the RFPE digital module 708 can comprise a plurality of processing boards, among which the image data is distributed; as an example, but without limitation, each processing board can receive an aggregate of image data at a rate of up to 50 Gbps; the extent of image processing conducted in the digital board can vary for each application; additional or alternative preprocessing steps can also be taken; and the processed image data is then stored in memory (e.g. memory 724) or data storage (e.g. data storage 728) for later deep learning with optimized selected vector concatenation (step 812), and See also at least ABSTRACT and Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 16-67 and Lines 1-16, Lines 64-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches the method and system, which has the digital focal plane array that includes the CMOS image sensor having the two-dimensional array of pixels, for obtaining and processing image data, by RFPE processor that includes one or more multi-threaded processors that perform image and graphical processing and provides images to a deep learning network using a highly parallel structure of data paths to manipulate large blocks of data, wherein instructions executable by the RFPE processor and the image data, obtained by the focal plane array, are stored in memory or data storage));
and an inference process performed using an artificial intelligence model on a basis of, as an input tensor, the image data for which image processing has been performed (738 FIGS. 7-8, Col. 10, Lines 1-45 of Schmidt teaches in processing actual or live image data, the process branches from step 816 to step 836, where the image data collected by the imaging system 108 is preprocessed and then processed using the OSVC model of the trained deep learning network 738; more particularly, the model can be used to return an output (step 840) in the form of a plurality of classified objects 120 corresponding to targets of interest to a user; the output results can thus include automated object classification, object identification to enable inferencing insights about target image objects 120 in a real image, and pose information to enable inferring further insights about target objects 120 in the real image; therefore, operation of a deployed network 704 can include identifying objects 120 within an image, and then selectively propagating pose information for some identified objects 120, and not propagating pose information for other identified objects 120; the operation of the deep learning network 738 can further include selecting a relevant set of images or areas within the full frame images that contain objects 120 of interest; moreover, the imaging system 108 can operate to identify those objects 120 of interest based on vector information, such as the pose of elements of an object within the scene or the relative pose of objects 120 to one another; other objects can be identified, but not associated with vector information that is propagated through the deep learning networks 738; for example, a lamp post may be identified by the deep learning network, but position and location information associated with the lamp post is not propagated through the deep learning network 738; in accordance with further embodiments of the present disclosure, the deep learning network 738 can select some or all images or sub-images containing objects of interest 120 for further analysis (i.e. potential autonomous action) or for delivery to a human analyst or to a system separate from the imaging system 108, for example via the communications interface 748 included as part of the host system 716; therefore, embodiments of the present disclosure can improve runtime performance of the deep learning network 738 itself; and moreover, the runtime performance of autonomous applications that utilize image information selected by the deep learning network 738 of communications networks associated with a communication interface 748, and/or the workload of human analyst or other systems can be improved, and See also at least Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 16-67 and Lines 1-16, Lines 17-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches the deep learning network implemented by the reconfigurable focal plane electronics (RFPE), which includes the RFPE processor that further includes the one or more multi-threaded processors that perform image and graphical processing and provides images to the deep learning network, that performs the processing of images that also includes training the deep learning network to predict a classification and identification of targets of interest and enable inferencing insights about target image objects in a real image, wherein during the training an HCN model is trained using images containing different poses of targets of interest and then optimized to form an OSVC model to subsequently automatically classify and identify objects in real image data).
Regarding claim 7, Schmidt teaches a program readable by a computer device, the program causing the computer device to implement: a function of storing image data output from a pixel array unit where multiple pixels are two-dimensionally arrayed; a function of causing execution of image processing for the stored image data (108, 704 FIGS. 7-8, Col. 7, Lines 17-63 of Schmidt teaches FIG. 8 depicts aspects of a deep learning network 738 in accordance with embodiments the present disclosure; the deep learning network 738 can be implemented by the RFPE 708 through execution of application instructions 736 stored in the RFPE memory 724 and executed by the RFPE processor 732; initially, at a data acquisition or input stage (step 804), an imaging system is operated to take multiple raw images of the same or different scenes 104; during a training phase, the images may contain objects 120 in the form of targets of interest for which automatic classification and identification by the deep learning network 738, after training, is desired; moreover, during training, the images may be obtained by an imaging system that is separate from and/or different than the imaging system 108; in accordance with other embodiments of the present disclosure, the imaging system 108 or an instance of the imaging system 108 that will be operated in connection with a trained deep learning network 738 as described herein can be used to obtain some or all of the images used for training; preprocessing is then performed (step 808); in preprocessing, some or all of the multiple images obtained as part of the data acquisition step are preprocessed to remove noise and image defects; more particularly, preprocessing can include frame co-addition, nonuniformity correction (NUC), compression, and data formatting processes; during a training phase, preprocessing may be performed using processing components that are different than and/or or provided separately from the RFPE 708; a reconfigurable focal plane electronics (RFPE) digital module 708 provided as part of an imaging system 108 can also be used to preprocess the multiple raw images to remove the different noise and image defects; moreover, in a deployed imaging system 108, the RFPE digital module 708 receives the multiple images obtained by the focal plane array 704 as part of the data acquisition step, and preprocesses those images to remove noise and image defects; in accordance with embodiments of the present disclosure, the RFPE digital module 708 can comprise a plurality of processing boards, among which the image data is distributed; as an example, but without limitation, each processing board can receive an aggregate of image data at a rate of up to 50 Gbps; the extent of image processing conducted in the digital board can vary for each application; additional or alternative preprocessing steps can also be taken; and the processed image data is then stored in memory (e.g. memory 724) or data storage (e.g. data storage 728) for later deep learning with optimized selected vector concatenation (step 812), and See also at least ABSTRACT and Col. 5, Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 60-67, Lines 1-67 and Lines 1-16, Lines 64-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches an imaging method and system, which has a digital focal plane array that includes a CMOS image sensor having a two-dimensional array of pixels, for obtaining and processing image data, by RFPE processor that includes one or more multi-threaded processors that perform image and graphical processing and provides images to a deep learning network using a highly parallel structure of data paths to manipulate large blocks of data, wherein instructions executable by the RFPE processor and the image data, obtained by the focal plane array, are stored in memory or data storage)); and a function of causing execution of an inference process that uses an artificial intelligence model on a basis of, as an input tensor, the image data for which image processing has been performed (738 FIGS. 7-8, Col. 10, Lines 1-45 of Schmidt teaches in processing actual or live image data, the process branches from step 816 to step 836, where the image data collected by the imaging system 108 is preprocessed and then processed using the OSVC model of the trained deep learning network 738; more particularly, the model can be used to return an output (step 840) in the form of a plurality of classified objects 120 corresponding to targets of interest to a user; the output results can thus include automated object classification, object identification to enable inferencing insights about target image objects 120 in a real image, and pose information to enable inferring further insights about target objects 120 in the real image; therefore, operation of a deployed network 704 can include identifying objects 120 within an image, and then selectively propagating pose information for some identified objects 120, and not propagating pose information for other identified objects 120; the operation of the deep learning network 738 can further include selecting a relevant set of images or areas within the full frame images that contain objects 120 of interest; moreover, the imaging system 108 can operate to identify those objects 120 of interest based on vector information, such as the pose of elements of an object within the scene or the relative pose of objects 120 to one another; other objects can be identified, but not associated with vector information that is propagated through the deep learning networks 738; for example, a lamp post may be identified by the deep learning network, but position and location information associated with the lamp post is not propagated through the deep learning network 738; in accordance with further embodiments of the present disclosure, the deep learning network 738 can select some or all images or sub-images containing objects of interest 120 for further analysis (i.e. potential autonomous action) or for delivery to a human analyst or to a system separate from the imaging system 108, for example via the communications interface 748 included as part of the host system 716; therefore, embodiments of the present disclosure can improve runtime performance of the deep learning network 738 itself; and moreover, the runtime performance of autonomous applications that utilize image information selected by the deep learning network 738 of communications networks associated with a communication interface 748, and/or the workload of human analyst or other systems can be improved, and See also at least Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 16-67 and Lines 1-16, Lines 17-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches the deep learning network implemented by the reconfigurable focal plane electronics (RFPE), which includes the RFPE processor that further includes the one or more multi-threaded processors that perform image and graphical processing and provides images to the deep learning network, that performs the processing of images that also includes training the deep learning network to predict a classification and identification of targets of interest and enable inferencing insights about target image objects in a real image, wherein during the training an HCN model is trained using images containing different poses of targets of interest and then optimized to form an OSVC model to subsequently automatically classify and identify objects in real image data).
Regarding claim 8, Schmidt teaches an image sensor comprising: a pixel array unit where multiple pixels are two-dimensionally arrayed (108 and 704 FIGS. 7-8, Cols. 5 and 6, Lines 60-67 and Lines 1-15, respectively of Schmidt teach FIG. 7 is a block diagram of an imaging system 108 in accordance with embodiments of the present disclosure; the imaging system 108 generally includes a digital focal plane array 704; the imaging system 108 also includes reconfigurable focal plane electronics (RFPE) 708; in accordance with at least some embodiments of the present disclosure, the RFPE 708 implements a microsequencer 712; the imaging system 708 can also include or can be connected to a host system 716; in addition, the imaging system can include a lens or optical assembly 720 for focusing collected light onto the focal plane array 704; as an example, but without limitation, the focal plane array 704 can include a backside illuminated CMOS image sensor having a 1024×1024 two-dimensional array of pixels; and as can be appreciated by one of skill in the art after consideration of the present disclosure, in addition to a focal plane array formed from a plurality of photosensitive sites or pixels, the focal plane array 708 can incorporate or be associated with driver and analog-to-digital conversion (ADC) circuitry, enabling the focal plane array 704 to provide a digital output representative of an amplitude or intensity of light detected at each pixel within the focal plane array 704, and See also at least ABSTRACT and Cols. 6 and 7, Lines 16-67 and Lines 1-16, respectively of Schmidt (i.e., Schmidt teaches an imaging method and system with a digital focal plane array that includes CMOS image sensor having a two-dimensional array of pixels));
an image processing unit that outputs second image data obtained by performing image processing for first image data (732 FIGS. 7-8, Col. 7, Lines 17-63 of Schmidt teaches FIG. 8 depicts aspects of a deep learning network 738 in accordance with embodiments the present disclosure; the deep learning network 738 can be implemented by the RFPE 708 through execution of application instructions 736 stored in the RFPE memory 724 and executed by the RFPE processor 732; initially, at a data acquisition or input stage (step 804), an imaging system is operated to take multiple raw images of the same or different scenes 104; during a training phase, the images may contain objects 120 in the form of targets of interest for which automatic classification and identification by the deep learning network 738, after training, is desired; moreover, during training, the images may be obtained by an imaging system that is separate from and/or different than the imaging system 108; in accordance with other embodiments of the present disclosure, the imaging system 108 or an instance of the imaging system 108 that will be operated in connection with a trained deep learning network 738 as described herein can be used to obtain some or all of the images used for training; preprocessing is then performed (step 808); in preprocessing, some or all of the multiple images obtained as part of the data acquisition step are preprocessed to remove noise and image defects; more particularly, preprocessing can include frame co-addition, nonuniformity correction (NUC), compression, and data formatting processes; during a training phase, preprocessing may be performed using processing components that are different than and/or or provided separately from the RFPE 708; a reconfigurable focal plane electronics (RFPE) digital module 708 provided as part of an imaging system 108 can also be used to preprocess the multiple raw images to remove the different noise and image defects; moreover, in a deployed imaging system 108, the RFPE digital module 708 receives the multiple images obtained by the focal plane array 704 as part of the data acquisition step, and preprocesses those images to remove noise and image defects; in accordance with embodiments of the present disclosure, the RFPE digital module 708 can comprise a plurality of processing boards, among which the image data is distributed; as an example, but without limitation, each processing board can receive an aggregate of image data at a rate of up to 50 Gbps; the extent of image processing conducted in the digital board can vary for each application; additional or alternative preprocessing steps can also be taken; and the processed image data is then stored in memory (e.g. memory 724) or data storage (e.g. data storage 728) for later deep learning with optimized selected vector concatenation (step 812), and See also at least ABSTRACT and Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 16-67 and Lines 1-16, Lines 64-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches the method and system, which has the digital focal plane array that includes the CMOS image sensor having the two-dimensional array of pixels, for obtaining and processing image data, by RFPE processor that includes one or more multi-threaded processors that perform image and graphical processing and provides images to a deep learning network using a highly parallel structure of data paths to manipulate large blocks of data, wherein instructions executable by the RFPE processor and the image data, obtained by the focal plane array, are stored in memory or data storage)); and an inference processing unit that performs an inference process for the second image data by using an artificial intelligence model and outputs an inference result, wherein the image processing uses multiple lines of the first image data for each processing (732, 738 FIGS. 7-8, Col. 10, Lines 1-45 of Schmidt teaches in processing actual or live image data, the process branches from step 816 to step 836, where the image data collected by the imaging system 108 is preprocessed and then processed using the OSVC model of the trained deep learning network 738; more particularly, the model can be used to return an output (step 840) in the form of a plurality of classified objects 120 corresponding to targets of interest to a user; the output results can thus include automated object classification, object identification to enable inferencing insights about target image objects 120 in a real image, and pose information to enable inferring further insights about target objects 120 in the real image; therefore, operation of a deployed network 704 can include identifying objects 120 within an image, and then selectively propagating pose information for some identified objects 120, and not propagating pose information for other identified objects 120; the operation of the deep learning network 738 can further include selecting a relevant set of images or areas within the full frame images that contain objects 120 of interest; moreover, the imaging system 108 can operate to identify those objects 120 of interest based on vector information, such as the pose of elements of an object within the scene or the relative pose of objects 120 to one another; other objects can be identified, but not associated with vector information that is propagated through the deep learning networks 738; for example, a lamp post may be identified by the deep learning network, but position and location information associated with the lamp post is not propagated through the deep learning network 738; in accordance with further embodiments of the present disclosure, the deep learning network 738 can select some or all images or sub-images containing objects of interest 120 for further analysis (i.e. potential autonomous action) or for delivery to a human analyst or to a system separate from the imaging system 108, for example via the communications interface 748 included as part of the host system 716; therefore, embodiments of the present disclosure can improve runtime performance of the deep learning network 738 itself; and moreover, the runtime performance of autonomous applications that utilize image information selected by the deep learning network 738 of communications networks associated with a communication interface 748, and/or the workload of human analyst or other systems can be improved, and See also at least Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 16-67 and Lines 1-16, Lines 17-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches the deep learning network implemented by the reconfigurable focal plane electronics (RFPE), which includes the RFPE processor that further includes the one or more multi-threaded processors that perform image and graphical processing and provides images to the deep learning network using the highly parallel structure of the data paths to manipulate the large blocks of data, that performs the processing of images and that also includes training the deep learning network to predict a classification and identification of targets of interest and enable inferencing insights about target image objects in a real image, wherein during the training an HCN model is trained using images containing different poses of targets of interest and then optimized to form an OSVC model to subsequently automatically classify and identify objects in real image data).
Regarding claim 9, Schmidt teaches an image sensor comprising: a pixel array unit where multiple pixels are two-dimensionally arrayed, the pixel array unit generating and outputting first image data (108 and 704 FIGS. 7-8, Cols. 5 and 6, Lines 60-67 and Lines 1-15, respectively of Schmidt teach FIG. 7 is a block diagram of an imaging system 108 in accordance with embodiments of the present disclosure; the imaging system 108 generally includes a digital focal plane array 704; the imaging system 108 also includes reconfigurable focal plane electronics (RFPE) 708; in accordance with at least some embodiments of the present disclosure, the RFPE 708 implements a microsequencer 712; the imaging system 708 can also include or can be connected to a host system 716; in addition, the imaging system can include a lens or optical assembly 720 for focusing collected light onto the focal plane array 704; as an example, but without limitation, the focal plane array 704 can include a backside illuminated CMOS image sensor having a 1024×1024 two-dimensional array of pixels; and as can be appreciated by one of skill in the art after consideration of the present disclosure, in addition to a focal plane array formed from a plurality of photosensitive sites or pixels, the focal plane array 708 can incorporate or be associated with driver and analog-to-digital conversion (ADC) circuitry, enabling the focal plane array 704 to provide a digital output representative of an amplitude or intensity of light detected at each pixel within the focal plane array 704, and See also at least ABSTRACT and Cols. 6 and 7, Lines 16-67 and Lines 1-16, respectively of Schmidt (i.e., Schmidt teaches an imaging method and system with a digital focal plane array that includes CMOS image sensor having a two-dimensional array of pixels that takes multiple raw images of different scenes containing objects of interest and their vector information));
an inference processing unit that performs an inference process for data based on the first image data by using an artificial intelligence model and outputs an inference result (732, 738 FIGS. 7-8, Col. 10, Lines 1-45 of Schmidt teaches in processing actual or live image data, the process branches from step 816 to step 836, where the image data collected by the imaging system 108 is preprocessed and then processed using the OSVC model of the trained deep learning network 738; more particularly, the model can be used to return an output (step 840) in the form of a plurality of classified objects 120 corresponding to targets of interest to a user; the output results can thus include automated object classification, object identification to enable inferencing insights about target image objects 120 in a real image, and pose information to enable inferring further insights about target objects 120 in the real image; therefore, operation of a deployed network 704 can include identifying objects 120 within an image, and then selectively propagating pose information for some identified objects 120, and not propagating pose information for other identified objects 120; the operation of the deep learning network 738 can further include selecting a relevant set of images or areas within the full frame images that contain objects 120 of interest; moreover, the imaging system 108 can operate to identify those objects 120 of interest based on vector information, such as the pose of elements of an object within the scene or the relative pose of objects 120 to one another; other objects can be identified, but not associated with vector information that is propagated through the deep learning networks 738; for example, a lamp post may be identified by the deep learning network, but position and location information associated with the lamp post is not propagated through the deep learning network 738; in accordance with further embodiments of the present disclosure, the deep learning network 738 can select some or all images or sub-images containing objects of interest 120 for further analysis (i.e. potential autonomous action) or for delivery to a human analyst or to a system separate from the imaging system 108, for example via the communications interface 748 included as part of the host system 716; therefore, embodiments of the present disclosure can improve runtime performance of the deep learning network 738 itself; and moreover, the runtime performance of autonomous applications that utilize image information selected by the deep learning network 738 of communications networks associated with a communication interface 748, and/or the workload of human analyst or other systems can be improved, and See also at least Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 16-67 and Lines 1-16, Lines 17-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches the deep learning network implemented by the reconfigurable focal plane electronics (RFPE), which includes the RFPE processor that further includes the one or more multi-threaded processors that perform image and graphical processing and provides images to the deep learning network using the highly parallel structure of the data paths to manipulate the large blocks of data, that performs the processing of images and that also includes continual training the deep learning network to predict a classification and identification of targets of interest and enable inferencing insights about target image objects in a real image, wherein during the training an HCN model is trained using images containing different poses of targets of interest and then optimized to form an OSVC model to subsequently automatically classify and identify objects in real image data); and an image processing unit that outputs second image data obtained by performing image processing for the first image data on a basis of the inference result, wherein the image processing uses multiple lines of the first image data for each processing (724, 732 FIGS. 7-8, Col. 7, Lines 17-63 of Schmidt teaches FIG. 8 depicts aspects of a deep learning network 738 in accordance with embodiments the present disclosure; the deep learning network 738 can be implemented by the RFPE 708 through execution of application instructions 736 stored in the RFPE memory 724 and executed by the RFPE processor 732; initially, at a data acquisition or input stage (step 804), an imaging system is operated to take multiple raw images of the same or different scenes 104; during a training phase, the images may contain objects 120 in the form of targets of interest for which automatic classification and identification by the deep learning network 738, after training, is desired; moreover, during training, the images may be obtained by an imaging system that is separate from and/or different than the imaging system 108; in accordance with other embodiments of the present disclosure, the imaging system 108 or an instance of the imaging system 108 that will be operated in connection with a trained deep learning network 738 as described herein can be used to obtain some or all of the images used for training; preprocessing is then performed (step 808); in preprocessing, some or all of the multiple images obtained as part of the data acquisition step are preprocessed to remove noise and image defects; more particularly, preprocessing can include frame co-addition, nonuniformity correction (NUC), compression, and data formatting processes; during a training phase, preprocessing may be performed using processing components that are different than and/or or provided separately from the RFPE 708; a reconfigurable focal plane electronics (RFPE) digital module 708 provided as part of an imaging system 108 can also be used to preprocess the multiple raw images to remove the different noise and image defects; moreover, in a deployed imaging system 108, the RFPE digital module 708 receives the multiple images obtained by the focal plane array 704 as part of the data acquisition step, and preprocesses those images to remove noise and image defects; in accordance with embodiments of the present disclosure, the RFPE digital module 708 can comprise a plurality of processing boards, among which the image data is distributed; as an example, but without limitation, each processing board can receive an aggregate of image data at a rate of up to 50 Gbps; the extent of image processing conducted in the digital board can vary for each application; additional or alternative preprocessing steps can also be taken; and the processed image data is then stored in memory (e.g. memory 724) or data storage (e.g. data storage 728) for later deep learning with optimized selected vector concatenation (step 812), and See also at least ABSTRACT and Cols. 6 and 7, Cols. 7 and 8, and Cols. 9 and 10; Lines 16-67 and Lines 1-16, Lines 64-67 and Lines 1-67, and Lines 1-67 and Lines 1-67, respectively of Schmidt (i.e., Schmidt teaches the method and system, which has the digital focal plane array that includes the CMOS image sensor having the two-dimensional array of pixels, capable of obtaining and processing image data, by RFPE processor that includes one or more multi-threaded processors that perform image and graphical processing and provides images to a deep learning network using a highly parallel structure of data paths to manipulate large blocks of data, wherein both instructions, executable by the RFPE processor and that implement the deep learning network to predict the classification and identification of targets of interest and enable inferencing insights about the target image objects in the real image, and the image data obtained by the focal plane array are stored in memory or data storage, and wherein the deep learning network selects some or all of the images containing objects of interest)).
Potentially Allowable Subject Matter
Claim 4 would be allowable if rewritten to overcome applicable objection(s) indicated above, and if rewritten in independent form including all of the limitations of the base claim and any intervening, because for claim 4 the prior art references of record do not teach the combination of all element limitations as presently claimed.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABDUL-SAMAD A ADEDIRAN whose telephone number is (571)272-3128. The examiner can normally be reached on Monday through Thursday, 8:00 am to 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amr Awad can be reached on 571-272-7764. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABDUL-SAMAD A ADEDIRAN/Primary Examiner, Art Unit 2621
Prosecution Insights