DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 6, & 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Balasubramanian (US 20190102640) in view of Terasaki (US 20200210818).
Regarding claim 1, Balasubramanian teaches an imaging device (Fig. 1 CNN system 100) including:
image sensor circuity that outputs first image data (¶33 An automated navigation system 100 for example can include one or more sensors 102 providing data sets as inputs to a CNN 104 that includes multiple different computational/processing layers with various components; ¶34 sensor 102 can include an image capturing device such as a camera, radar, light detection and ranging (LIDAR), other image scanning devices, or other sensors for detecting and processing data from images, such as those received on the signals from an image capturing device);
first processing circuity that executes processing of a first layer in a neural network having a layered structure (¶39 The CNN 104 can comprise a number of computational layers, including a convolution layer 202, a rectified linear unit (RELU) layer 204, a pooling layer 206, a fully connected (FC) layer 208 (artificial neural network layer), and an output layer 210) on the first image data in units of second image data having a size smaller than an entire size of the first image data (¶29 a convolution sliding window (e.g., a 3×3 convolution sliding window or other size of n x n convolution sliding window, with n as a positive integer) can be selected to perform convolution of an image that has been received from sensor data retrieved, and then multiplied with the filter kernel as an inner dot product operation; ¶40 The convolution processes can be performed on sets/segments/subsets/portions of the image data, for example, along sections of an image 232 for a particular feature); and
second processing circuitry that executes processing of a second layer in the neural network on a processing result output from the first processing circuitry (¶59 Execution pipelining or pipeline processing can be referred to herein as a set of data processing elements, components or functions connected in series, where the output of one component or element is the input of the next one).
Balasubramanian does not teach where Terasaki teaches wherein the first processing circuitry executes the processing of the first layer in synchronization with output of data of a predetermined number of lines from the image sensor circuitry (¶241 The synchronous array unit 241 of the first layer receives and processes a signal output from the device interface 221 and outputs the processed signal to the synchronous array unit 242 of the second layer); and
second processing circuitry that executes processing of a second layer in the neural network (Fig. 4 neural network system 201) on a processing result output from the first processing circuitry in synchronization with output of the processing result from the first processing circuitry (¶242 The synchronous array unit 242 of the second layer receives and processes a signal output from the synchronous array unit 241 of the first layer and outputs the processed signal to the identification unit 232).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of wherein the first processing circuitry executes the processing of the first layer in synchronization with output of data of a predetermined number of lines from the image sensor circuitry; and second processing circuitry that executes processing of a second layer in the neural network on a processing result output from the first processing circuitry in synchronization with output of the processing result from the first processing circuitry from Terasaki into the imaging device as disclosed by Balasubramanian. The motivation for doing this is to improve neural network processing by using low power at a high speed.
Regarding claim 2, Balasubramanian teaches the imaging device according to claim 1, wherein the second image data is image data for a predetermined number of lines in the first image data (¶68 Further utilizing the 3×3 window 308, small sized image data portions that are size 3×3 as an example can be analyzed from the image data stored in memory 304; Sliding the window over by another column, the window results can be 3, 4, 5, 9, 1, 3, 5, 9, 3, forming another sliding window result; e.g. a sliding convolution window necessarily operates on a limited set of image rows and columns, corresponding to a predetermined number of lines).
Regarding claim 3, Balasubramanian teaches the imaging device according to claim 1, wherein the processing of the first layer includes convolution processing (¶40 The convolution layer 202, for example, can include one or more convolution components 212 that extract data slices of an image 232 as data sets. The convolution layer 202 can be combined with the rectified linear unit (RELU) layer 204 to also be considered or referred to as one computational layer 230, or, in general, as a convolution layer 230).
Regarding claim 4, Balasubramanian teaches the imaging device according to claim 3, wherein the second image data is image data for a number of lines corresponding to the number of rows of a filter used in the convolution processing (¶29 a convolution sliding window (e.g., a 3×3 convolution sliding window or other size of n x n convolution sliding window, with n as a positive integer) can be selected to perform convolution of an image that has been received from sensor data retrieved, and then multiplied with the filter kernel as an inner dot product operation; ¶68 Sliding the window over by another column, the window results can be 3, 4, 5, 9, 1, 3, 5, 9, 3, forming another sliding window result).
Regarding claim 6, Balasubramanian teaches the imaging device according to claim 3, wherein the processing of the second layer includes full-connection processing (¶39 The CNN 104 can comprise a number of computational layers, including a convolution layer 202, a rectified linear unit (RELU) layer 204, a pooling layer 206, a fully connected (FC) layer 208 (artificial neural network layer), and an output layer 210).
Regarding claim(s) 15 (drawn to a method):
The rejection/proposed combination of Balasubramanian and Terasaki, explained in the rejection of device claim(s) 1, anticipates/renders obvious the steps of the method of claim(s) 15 because these steps occur in the operation of the proposed combination as discussed above. Thus, the arguments similar to that presented above for claim(s) 1 is/are equally applicable to claim(s) 15.
Regarding claim(s) 16 (drawn to an imaging program):
The rejection/proposed combination of Balasubramanian and Terasaki, explained in the rejection of device claim(s) 1, anticipates/renders obvious the steps of the imaging program of claim(s) 16 because these steps occur in the operation of the proposed combination as discussed above. Thus, the arguments similar to that presented above for claim(s) 1 is/are equally applicable to claim(s) 16. See further Balasubramanian ¶134-135.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Balasubramanian and Terasaki as applied to claim 4 above, and further in view of Kondo et al (US 20120182321).
Regarding claim 5, the combination of Balasubramanian and Terasaki teaches the imaging device according to claim 4, but fails teach where Kondo teaches further including a line memory for the number of lines corresponding to the number of rows of the filter (¶92 The line memory 102 need only be capable of storing as many rows of pixel data as the number of taps of the interpolation filter 103), the image sensor circuitry sequentially reads the first image data in units of lines and inputs the first image data to the line memory (¶360 the pixel data is sequentially read into the line memory 22 in the order in which the partial image is raster scanned), and the first processing circuitry executes the processing of the first layer on the second image data stored in the line memory (¶360 the pixel data is sequentially read into the line memory 22 in the order in which the partial image is raster scanned; when the enlarging or reducing processing of a next row begins, the rows of data that were read in the past remain stored in the line memory 22).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of further including a line memory for the number of lines corresponding to the number of rows of the filter, the image sensor circuitry sequentially reads the first image data in units of lines and inputs the first image data to the line memory, and the first processing circuitry executes the processing of the first layer on the second image data stored in the line memory from Kondo into the imaging device as disclosed by Balasubramanian and Terasaki. The motivation for doing this is to improve image conversion method, program and electronic equipment for enlarging or reducing an image.
Claim(s) 7-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Balasubramanian and Terasaki as applied to claim 1 and 3 above, and further in view of Han et al (US 20190251694).
Regarding claim 7, the combination of Balasubramanian and Terasaki teaches the imaging device according to claim 3, but fails teach where Han teaches wherein the processing of the second layer includes deconvolution processing (¶69 one or more deconvolutional layers).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of wherein the processing of the second layer includes deconvolution processing from Han into the imaging device as disclosed by the combination of Balasubramanian and Terasaki. The motivation for doing this is to improve imaging techniques such as image segmentation with the integration of deep learning.
Regarding claim 8, the combination of Balasubramanian and Terasaki teaches the imaging device according to claim 1, but fails to teach where Han teaches wherein at least one of the processing of the first layer and the processing of the second layer includes enlargement processing of enlarging the second image data (¶69 one or more deconvolutional layers; ¶87 deconvolution network 434 enlarges the intermediate activation maps or feature maps by using a selection of deconvolutional layers 436 and/or unpooling layers (not shown)).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of wherein at least one of the processing of the first layer and the processing of the second layer includes enlargement processing of enlarging the second image data from Han into the imaging device as disclosed by the combination of Balasubramanian and Terasaki. The motivation for doing this is to improve imaging techniques such as image segmentation with the integration of deep learning.
Regarding claim 9, the combination of Balasubramanian and Terasaki teaches the imaging device according to claim 1, but fails to teach where Han teaches wherein at least one of the processing of the first layer and the processing of the second layer includes interpolation processing of interpolating between pixels in the second image data (¶86 various functions may be used to implement the pixel-wise prediction layer, such as backwards upsampling or unpooling (e.g., bilinear or nonlinear interpolation)).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of wherein at least one of the processing of the first layer and the processing of the second layer includes interpolation processing of interpolating between pixels in the second image data from Han into the imaging device as disclosed by the combination of Balasubramanian and Terasaki. The motivation for doing this is to improve imaging techniques such as image segmentation with the integration of deep learning.
Claim(s) 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Balasubramanian and Terasaki as applied to claim 1 above, and further in view of Kouada et al (US Patent 9588240 B1).
Regarding claim 10, the combination of Balasubramanian and Terasaki teaches the imaging device according to claim 1, but fails to teach where Kouada teaches wherein the image sensor circuitry, the first processing circuitry, and the second processing circuitry are mounted on a single chip (col 1 lines 5-27 CMOS image sensors can be integrated with all kinds of functional circuitry and blocks in a single chip; A digital imager typically can include a photodiode array, column readout structure, A/D conversion, and digital controllers (or processors) on single or multiple substrates; FIG. 1 depicts conventional four-side buttable BSI imager 100 using multiple layers of chips stacked in a three-dimensional (3D) package. On a first layer, the 3D BSI imager includes imaging sensor array 110 with pixels containing photodiodes that are exposed to incident light).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of wherein the image sensor circuitry, the first processing circuitry, and the second processing circuitry are mounted on a single chip from Kouada into the imaging device as disclosed by the combination of Balasubramanian and Terasaki. The motivation for doing this is to improve performance of image sensors.
Regarding claim 11, the combination of Balasubramanian, Terasaki and Kouada teaches the imaging device according to claim 10, wherein the single chip is a stacked chip, and the image sensor circuitry is disposed in a first layer in the stacked chip, and at least one of the first processing circuitry and the second processing circuitry is disposed in a second layer in the stacked chip (Kouada col 1 lines 5-27 CMOS image sensors can be integrated with all kinds of functional circuitry and blocks in a single chip; A digital imager typically can include a photodiode array, column readout structure, A/D conversion, and digital controllers (or processors) on single or multiple substrates; FIG. 1 depicts conventional four-side buttable BSI imager 100 using multiple layers of chips stacked in a three-dimensional (3D) package. On a first layer, the 3D BSI imager includes imaging sensor array 110 with pixels containing photodiodes that are exposed to incident light). The motivation to combine the references is discussed above in the rejection for claim 10.
Claim(s) 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Balasubramanian and Terasaki as applied to claim 4 above, and further in view of Sonoda (US 20190305056).
Regarding claim 17, the combination of Balasubramanian and Terasaki teaches the imaging device according to claim 4, but fails to teach where Sonoda teaches a line memory for the number of lines corresponding to the number of rows of the filter (¶56 When the linear filter that uses the seven filter coefficients is used, the region divider may include the seven line memories for seven rows).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of a line memory for the number of lines corresponding to the number of rows of the filter from Sonoda into the imaging device as disclosed by the combination of Balasubramanian and Terasaki. The motivation for doing this is to improve methods and devices that divides an image into a plurality of regions and respectively compensates the image of the regions is used in order to improve display quality.
Regarding claim 17, the combination of Balasubramanian and Terasaki teaches the imaging device according to claim 4, but fails to teach where Sonoda teaches a line memory for the number of lines corresponding to the number of rows of the filter (¶56 When the linear filter that uses the seven filter coefficients is used, the region divider may include the seven line memories for seven rows), and the image sensor circuitry sequentially reads the first image data in units of lines and inputs the first image data to the line memory (¶56 The region divider may read the image data from the most recent pixel row as the row number of the line memory and input the image data of the next row into the last row of the line memory whenever the process of one row is finished).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of a line memory for the number of lines corresponding to the number of rows of the filter from Sonoda into the imaging device as disclosed by the combination of Balasubramanian and Terasaki. The motivation for doing this is to improve methods and devices that divides an image into a plurality of regions and respectively compensates the image of the regions is used in order to improve display quality.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-11 and 15-18 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEVIN KY whose telephone number is (571)272-7648. The examiner can normally be reached Monday-Friday 9-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEVIN KY/ Primary Examiner, Art Unit 2671
Prosecution Insights