DETAILED ACTION
Claims 1-20 are pending.
Claims 1, 8 and 14 are independent claims.
Claims 1, 4, 8, 12 and 14 are amended.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered. However, Jin and Ling has been remapped in combination with references Avital and Kim to teach the amended features. Examiner respectfully directs Applicant to the detailed rejection for an explanation of how the references disclose the argued limitations.
Applicant argues:
The Applicant has amended independent Claim 1 to recite, among other things, "concatenating, by the processor-based system, the first group of pooled feature channels generated by the first windowed pooling process with the second group of pooled feature channels generated by the second windowed pooling process to generate merged pooled feature channels".
In Jin, paragraph 0046 teaches input data being put into multiple channels and being processed through pooling processes to merge the channels together. (Jin, paragraph 0046, “Referring to FIG. 2A, input data 200 on the leftmost side may include a plurality of channels. FIG. 2 shows an example in which the input data 200 includes three channels [generate merged pooled feature channels;, examiner would like to point out that the channels the input is put into the convolutional layers at the same time, being interpreted as merging them together.]. The input data 200 may be expressed in width, height and depth. Details of input data 210 with one channel are shown in terms of width, height and length. In FIGS. 2A and 2B, 210 to 240 show an example of data processed by a convolution operation and pooling operation with respect to data inputted to one channel. Data inputted to other channels may be processed using the same method [the first group of pooled feature channels generated by the first window pooling process with the second group of pooled feature channels]. Input data 200 may be zero-padded in order to comply with the size of a convolution window. Zero padding may mean that 0 is inserted into the edge of the input data 210.”)
Applicant also argues:
Kim discloses a method for processing a convolutional neural network (CNN) using a systolic array wherein the operational result for one layer is used as an input to the operation for a next layer. More specifically, Kim discloses a method wherein each layer of a CNN processor generates M output feature maps using N input feature maps.6 The value of an output pixel at a particular position in the output feature map is determined by applying a three-dimensional weight of K x K x N around the adjacent input pixels at corresponding positions of the N input feature maps.7 When the three-dimensional weight of K x K x N is applied around adjacent input pixels, the result in that a given pixel would be processed more than once as neighboring pixels are processed. This is an express teaching away from the claimed requirement that each pooled feature channel "was generated by exactly one" of the first and second windowed pooling processes (in Claim 8) or "has been processed by exactly one" of the first and second operations (in Claim 14).
Claim 8 has the limitation “wherein each pooled feature channel in an aggregated group of the first group of pooled feature channels and the second group of pooled feature channels was generated by exactly one of the first windowed pooling process or the second windowed pooling process…” In Kim, paragraph 0049 discusses the how with the feature channels that the CNN processor may perform pooling for a given window size. (Kim, paragraph 0048, “That is, the value of the output pixel at a particular position in an output feature map is determined by applying a three-dimensional weight of K*K*N around the adjacent input pixels at the corresponding positions of the N input feature maps, the input feature map is multiplied by the values of the input pixels, added together, and then added together with the bias corresponding to the output feature map [the first group of pooled feature channels and the second group of pooled feature channels, examiner interprets the input feature map as the first group of pooled feature channels and then the output feature map is interpreted as the second group of feature channels.].” and paragraph 0049, “In addition, the CNN processor may perform pooling after such convolution and activation, for example, by selecting the largest value for a given window size, for example, a 2*2 window, or by reducing the size of the feature map. Depending on the implementation, convolution, batch normalization, activation, and pooling may be called individual layers, or a combination of several thereof may be defined as one layer.”). Examiner would like to point out that the pooling process is being interpreted as the first operation for the feature channels. Examiner would like to also point out that in the abstract of Kim the description discusses the one pooling process and it happening to a plurality of adjacent pixels.
As for claim 14, the limitation states, “wherein each pooled feature channel in an aggregated group of the first group of pooled feature channels and the second group of pooled feature channels was generated by exactly one of the first operations or the second operation…” In Kim, paragraph 0049 discusses the how with the feature channels that the CNN processor may perform pooling for a given window size. (Kim, paragraph 0048, “That is, the value of the output pixel at a particular position in an output feature map is determined by applying a three-dimensional weight of K*K*N around the adjacent input pixels at the corresponding positions of the N input feature maps, the input feature map is multiplied by the values of the input pixels, added together, and then added together with the bias corresponding to the output feature map [the first group of pooled feature channels and the second group of pooled feature channels, examiner interprets the input feature map as the first group of pooled feature channels and then the output feature map is interpreted as the second group of feature channels.].” and paragraph 0049, “In addition, the CNN processor may perform pooling after such convolution and activation, for example, by selecting the largest value for a given window size, for example, a 2*2 window, or by reducing the size of the feature map. Depending on the implementation, convolution, batch normalization, activation, and pooling may be called individual layers, or a combination of several thereof may be defined as one layer.”). Examiner would like to point out that the pooling process is being interpreted as the first operation for the feature channels. Examiner would like to also point out that in the abstract of Kim the description discusses the one pooling process and it happening to a plurality of adjacent pixels. Therefore, the 35 USC 103 rejection is upheld.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 12 rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. Claim 12 states "The system of claim 8, wherein the one or more processors are further configured to apply the contextually pooled output feature channels to an output CNN to qenerate a detection and/or a class prediction for one or more objects in the input image." This is the same as the last limitation of claim 8, making it so dependent claim 12 does not further limit the independent claim.. Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 4-6, 14-15 and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Jin et al (US Published Patent Application No. 20180089562, "Jin"), in view of Avital et al (US Published Patent Application No. 20200380687, "Avital") and Kim et al (US Published Patent Application No. 20190164037, “Kim”)
In regard to claim 1, Jin teaches applying, by the processor-based system, a first windowed pooling process to the first group of feature channels, (Jin, paragraph 0043, Jin teaches a first windowed pooling process to the first group of feature channels. The pooling process happens in the feature extraction part in the pooling layer. The data output by the first convolutional layer is interpreted as the first group of feature channels.)
and wherein applying the first windowed pooling process generates a first group of pooled feature channels; (Jin, paragraph 0043, “The feature extraction part 110 may include an architecture in which output data processed by a pair of a convolution layer and a pooling layer becomes the input data of a next pair of a convolution layer and a pooling layer. For example, external input data 100 may become the input data of the first convolution layer of the feature extraction part 110, and data processed and output by the first convolution layer may become the input data of the first pooling layer.” Also see 0046 and 0048).
applying, by the processor-based system, a second windowed pooling process to the second group of feature channels, (Jin, paragraph 0043, Jin teaches the second window pooling process on the second group of feature channels. Jin discusses the pooling process being performed as many times as set. Allowing for the second pooling process to be done on the second group of feature channels. Also see 0046 and 0048)
and wherein applying the second windowed pooling process generates a second group of pooled feature channels; (Jin, paragraph 0043, “Furthermore, data processed by the first pooling layer [generates a second group of pooled feature channels, examiner interprets the pooling layer as the second pooled feature channel due to the pooling operation being able to perform as many times as necessary]] may become the input data of a second convolution layer, that is, one of a next pair. A convolution operation and a pooling operation may be performed by the number of times [wherein applying the second windowed pooling] (e.g., N times) set in a host.”)
concatenating, by the processor-based system, the first group of pooled feature channels generated by the first window pooling process with the second group of pooled feature channels generated by the second window pooling process to generate merged pooled feature channels; (Jin, paragraph 0046, “Referring to FIG. 2A, input data 200 on the leftmost side may include a plurality of channels. FIG. 2 shows an example in which the input data 200 includes three channels [generate merged pooled feature channels;, examiner would like to point out that the channels the input is put into the convolutional layers at the same time, being interpreted as merging them together]. The input data 200 may be expressed in width, height and depth. Details of input data 210 with one channel are shown in terms of width, height and length. In FIGS. 2A and 2B, 210 to 240 show an example of data processed by a convolution operation and pooling operation with respect to data inputted to one channel [concatenating, examiner would like to point out that these being inputted into one channel is linking them together]. Data inputted to other channels may be processed using the same method [the first group of pooled feature channels generated by the first window pooling process with the second group of pooled feature channels]. Input data 200 may be zero-padded in order to comply with the size of a convolution window. Zero padding may mean that 0 is inserted into the edge of the input data 210.”)
concatenating, by the processor-based system, the merged pooled feature channels with the input feature channels to generate concatenated feature channels; and (Jin, paragraph 0046, “Referring to FIG. 2A, input data 200 on the leftmost side may include a plurality of channels. FIG. 2 shows an example in which the input data 200 includes three channels. The input data 200 may be expressed in width, height and depth. Details of input data 210 with one channel are shown in terms of width, height and length. In FIGS. 2A and 2B, 210 to 240 show an example of data processed by a convolution operation and pooling operation with respect to data inputted to one channel. Data inputted to other channels may be processed using the same method. Input data 200 may be zero-padded in order to comply with the size of a convolution window [input feature channels to generate concatenated feature channels]. Zero padding may mean that 0 is inserted into the edge of the input data 210.”)
applying, by the processor-based system, a two-dimensional convolutional neural network (CNN) to the concatenated feature channels to generate contextually pooled output feature channels. (Jin, paragraph 0052, “Referring to FIG. 3, in an initialization operation 310, the CNN operation apparatus may receive and store pieces of information for executing a CNN algorithm. The pieces of received information may become weight data. Furthermore, the CNN operation apparatus may further receive and store at least one of layer configuration information, an address map for convolution input (or a precomputed memory map) and an activation function look-up table (LUT). After performing the initialization operation 310, the CNN operation apparatus may perform a feature extraction operation [a two-dimensional convolutional neural network (CNN) to the concatenated feature channels] 320. The feature extraction operation 320 may become a convolution operation and a pooling operation. The convolution operation and the pooling operation may be performed according to a method and procedure, such as those shown in FIGS. 1 to 2B. When completing the feature extraction operation 320 for the input data, the CNN operation apparatus may perform a classification operation 330. The classification operation 330 may include at least one fully connected layer as shown in FIG. 1. When completing the classification operation 330, the CNN operation apparatus may output data to the outside of a system in a finish operation [generate contextually pooled output feature channels. ] 340.”)
However, Jin does not explicitly teach segmenting, by a processor-based system, the plurality of input feature channels into a first group of feature channels and a second group of feature channels;
Avital teaches segmenting, by a processor-based system, the plurality of input feature channels into a first group of feature channels and a second group of feature channels; (Avital, paragraph 0158 and paragraph 0159, “The number of feature channels doubles for each sequential stage of the encoding-contracting path, and is halved for each sequential stage of the decoding-expanding path [plurality of input feature channels into a first group of feature channels and a second group of feature channels;]. For example, the number of 3D voxel filters (each corresponding to a respective feature channel) at each stage of each instance of the common component are: I 0, 20, 40, 80, and 160.”)
Jin and Avital are related to the same field of endeavor (i.e. convolutional neural networks). In view of the teachings of Avital, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Avital to Jin before the effective filing date of the claimed invention in order to improve the ability to automatically segment predefined regions. (Avital, paragraph 0014, “The systems, apparatus, methods and/or code instructions (stored in a data storage device, executable by one or more hardware processors) described herein are directed to an improvement in computer-related technology, by improving the ability of computers to automatically segment predefined anatomical regions based on multimodal images.”)
However, Jin and Avital do not explicitly teach receiving, by a processor-based system, an input image comprising an array of pixels, at least a portion of the pixels having assigned thereto a pixel value;
extracting, by the processor-based system, a plurality of input feature channels from the input image, wherein each input feature channel is a sub-array of pixels extracted from the input image;
wherein the first windowed pooling process applies a first operation to a plurality of horizontally adjacent pixels in each feature channel in the first group of feature channels,
wherein the second windowed pooling process applies a second operation to a plurality of vertically adjacent pixels in each feature channel in the second group of feature channels,
wherein each pooled feature channel in an aggregated group of the first group of pooled feature channels and the second group of pooled feature channels has been processed by exactly one of the first operation or the second operation.
Kim teaches receiving, by a processor-based system, an input image comprising an array of pixels, at least a portion of the pixels having assigned thereto a pixel value; (Kim, paragraph 0004, Kim teaches generating feature maps from an input image. And paragraph 0019, Kim teaches using a processor to generate an output feature map. Each value in the feature map correspond to an output pixel, determined by applying a three-dimensional weight of K*K*N, as shown in paragraph 0048. Since the values in the feature map are associated with specific pixels, Kim teaches determining the pixel values.)
extracting, by the processor-based system, a plurality of input feature channels from the input image, wherein each input feature channel is a sub-array of pixels extracted from the input image; (Kim, paragraph 0011, Kim teaches obtaining address information of the feature map and the plurality of input pixels contained in the input feature map. The foregoing teaches extracting a plurality of input feature channels from the input image.)
wherein the first windowed pooling process applies a first operation to a plurality of horizontally adjacent pixels in each feature channel in the first group of feature channels, (Kim, paragraph 0048, “That is, the value of the output pixel at a particular position in an output feature map is determined by applying a three-dimensional weight of K*K*N around the adjacent input pixels [adjacent pixels in each feature channel in the first group of feature channels] at the corresponding positions of the N input feature maps, the input feature map is multiplied by the values of the input pixels, added together, and then added together with the bias corresponding to the output feature map.” and paragraph 0112, “The code below represents a method of generating an address of a scheme including steps of processing the coordinates of the output feature map vertically and horizontally, processing pooling positions [first windowed pooling process applies a first operation] of vertical and horizontal directions [a plurality of horizontally adjacent pixels] in itself, processing K*K weights for each value, and processing in a channel direction initially for each weight position (in the manner of processing for each of K*K by direction of N).”)
wherein the second windowed pooling process applies a second operation to a plurality of vertically adjacent pixels in each feature channel in the second group of feature channels, (Kim, paragraph 0048, “That is, the value of the output pixel at a particular position in an output feature map is determined by applying a three-dimensional weight of K*K*N around the adjacent input pixels [adjacent pixels in each feature channel in the second group of feature channels] at the corresponding positions of the N input feature maps, the input feature map is multiplied by the values of the input pixels, added together, and then added together with the bias corresponding to the output feature map.” and paragraph 0112, “The code below represents a method of generating an address of a scheme including steps of processing the coordinates of the output feature map vertically and horizontally, processing pooling positions [the second windowed pooling process applies a second operation] of vertical [a plurality of vertically adjacent pixels] and horizontal directions in itself, processing K*K weights for each value, and processing in a channel direction initially for each weight position (in the manner of processing for each of K*K by direction of N).”)
wherein each pooled feature channel in an aggregated group of the first group of pooled feature channels and the second group of pooled feature channels has been processed by exactly one of the first operation or the second operation; (Kim, paragraph 0048, “That is, the value of the output pixel at a particular position in an output feature map is determined by applying a three-dimensional weight of K*K*N around the adjacent input pixels at the corresponding positions of the N input feature maps, the input feature map is multiplied by the values of the input pixels, added together, and then added together with the bias corresponding to the output feature map [the first group of pooled feature channels and the second group of pooled feature channels, examiner interprets the input feature map as the first group of pooled feature channels and then the output feature map is interpreted as the second group of feature channels.].” and paragraph 0049, In addition, the CNN processor may perform pooling after such convolution and activation, for example, by selecting the largest value for a given window size, for example, a 2*2 window, or by reducing the size of the feature map [has been processed by exactly one of the first operation]. Depending on the implementation, convolution, batch normalization, activation, and pooling may be called individual layers, or a combination of several thereof may be defined as one layer.” And paragraph 0112, “The code below represents a method of generating an address of a scheme including steps of processing the coordinates of the output feature map vertically and horizontally, processing pooling positions of vertical and horizontal directions in itself, processing K*K weights for each value, and processing in a channel direction initially for each weight position (in the manner of processing for each of K*K by direction of N).”)
Jin, Avital and Kim are related to the same field of endeavor (i.e. convolutional neural networks). In view of the teachings of Kim, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Kim to Jin and Avital before the effective filing date of the claimed invention in order to efficiently store the input and output feature maps. (Kim, paragraph 0008, “Embodiments of the present invention provide an apparatus for processing a convolutional neural network using a systolic array and a method thereof using the operational result for one layer as an input to the operation for a next layer, while using the systolic array easily, and efficiently storing an input feature map and an output feature map.”)
In regard to claim 2 and analogous claim 15, Jin, Avital and Kim teach the method of claim 1.
Jin further teaches wherein the first windowed pooling process is a maximum pooling process or a minimum pooling process, and the second windowed pooling process is a mean pooling process. (Jin, paragraph 0048, “When the convolution operation is performed, output data, such as data 230 of FIG. 2B, may be generated. When the convolution operation is terminated, the output data may be computed as an activation value through one of activation functions 240. In FIG. 2B, "249" shows an example in which an activation function is expressed in equation. An activation output value may become the input data of the pooling layer 250. A pooling operation may be an operation for selecting a maximum value of the values of a plurality of input data that enters a pooling window [maximum pooling process], such as a polling window 241 or 242 in FIG. 2B, or calculating an average value of the plurality of input data [a mean pooling process]. Such maximum values or average values may be compressed into one value, such as 251 or 252. In FIG. 2B, "259" shows an example in which the pooling operation is expressed in equation.”)
In regard to claim 4, Jin, Avital and Kim teach the method of claim 1.
Jin further teaches wherein concatenating the first group of pooled feature channels with the second group of pooled feature channels is performed with weighting factors generated by a gate selection CNN. (Jin, paragraph 0046, “Referring to FIG. 2A, input data 200 on the leftmost side may include a plurality of channels. FIG. 2 shows an example in which the input data 200 includes three channels. The input data 200 may be expressed in width, height and depth. Details of input data 210 with one channel are shown in terms of width, height and length. In FIGS. 2A and 2B, 210 to 240 show an example of data processed by a convolution operation and pooling operation with respect to data inputted to one channel[concatenating, examiner would like to point out that these being inputted into one channel is linking them together]. Data inputted to other channels may be processed using the same method [the first group of pooled feature channels generated by the first window pooling process with the second group of pooled feature channels]. Input data 200 may be zero-padded in order to comply with the size of a convolution window. Zero padding may mean that 0 is inserted into the edge of the input data 210.” And paragraph 0122, “An OR gate 820 may perform OR operation on a Z_F, included in an input data set having the same architecture as the input data set 911, and a Z_F Included in a weight data set having the same architecture as the weight data set 913 [weighting factors generated by a gate selection CNN]. A selector (or multiplexer) 830 may first input the output of the multiplier 810 (i.e., the first input), may second input a ground (or zero) signal (i.e., the second input), may select and out one of the first input or the second input based on the output of the OR gate 820. Each of the PEs of the PE unit 545 may include the multiplier 810 corresponding to the number of points of input data. For example, each PE may be a 16-bit or 32-bit/floating or fixed point multiplier. Each PE may receive an input data set and a weight data set, such as those shown in FIG. 9A, under the control of the PE controller 543. The multiplier 810 may perform convolution multiplication operation on the received two data.”)
In regard to claim 5 and analogous claim 18, Jin, Avital and Kim teach the method of claim 1.
JIn further teaches wherein the plurality of input feature channels are extracted by applying backbone CNN applied to an input image. (Jin, paragraph 0149, “As shown in FIG. 3, the CNN operation apparatus in accordance with various embodiments of the present invention may perform a feature extraction layer operation 320 on the input data of all of channels after performing an initialization operation 310. [plurality of input feature channels are extracted by applying backbone CNN applied to an input image.] When the feature extraction layer operation 320 is completed, the CNN operation apparatus may perform a classification layer operation 330. When the classification layer operation 330 is completed, the CNN operation apparatus may send a learnt result value to the host.”)
In regard to claim 6 and analogous claim 19, Jin, Avital and Kim teach the method of claim 1.
Jin further teaches wherein the plurality of input feature channels are extracted from the input image using a contextual pooling process. (Jin, paragraph 0071, “In the CNN operation apparatus having a configuration, such as that described above, when first executing a feature extraction layer operation, the CNN control unit 405 may store data, received from an external system, in the input unit 435 of the feature extraction layer block 425. When executing feature extraction layers subsequent to the first feature extraction layer, the CNN control unit 405 may remap the output data of the pooling block 455 stored in the output unit 465 and may store the remapped data in the input unit 435 as convolution input data [input feature channels are extracted…a contextual pooling process.]. When the feature extraction layer operation is terminated, the CNN control unit 405 may execute the classification layer block 427. The classification layer block 427 may remap output data computed by the convolution block 447 and may store the remapped data in the input unit 437 as the input data of a fully connected layer [extracted from the input image]. When the classification layer operation is terminated, the CNN control unit 405 may output the resulting data to an external system.”)
In regard to claim 14, Jin teaches applying a first windowed pooling process to the first group of feature channels, (Jin, paragraph 0043, Jin teaches a first windowed pooling process to the first group of feature channels. The pooling process happens in the feature extraction part in the pooling layer. The data output by the first convolutional layer is interpreted as the first group of feature channels.)
and wherein applying the first windowed pooling process generates a first group of pooled feature channels; (Jin, paragraph 0043, “The feature extraction part 110 may include an architecture in which output data processed by a pair of a convolution layer and a pooling layer becomes the input data of a next pair of a convolution layer and a pooling layer. For example, external input data 100 may become the input data of the first convolution layer of the feature extraction part 110, and data processed and output by the first convolution layer may become the input data of the first pooling layer.” Also see 0046 and 0048).
applying a second windowed pooling process to the second group of feature channels, (Jin, paragraph 0043, Jin teaches the second window pooling process on the second group of feature channels. Jin discusses the pooling process being performed as many times as set. Allowing for the second pooling process to be done on the second group of feature channels.)
wherein applying the second windowed pooling process generates a second group of pooled feature channels, (Jin, paragraph 0043, “Furthermore, data processed by the first pooling layer [generates a second group of pooled feature channels, examiner interprets the pooling layer as the second pooled feature channel due to the pooling operation being able to perform as many times as necessary]] may become the input data of a second convolution layer, that is, one of a next pair. A convolution operation and a pooling operation may be performed by the number of times [wherein applying the second windowed pooling] (e.g., N times) set in a host.” Also see 0046 and 0048)
performing a merging of the first group of pooled feature channels and the second group of pooled feature channels to generate merged pooled feature channels; (Jin, paragraph 0046, “Referring to FIG. 2A, input data 200 on the leftmost side may include a plurality of channels. FIG. 2 shows an example in which the input data 200 includes three channels [generate merged pooled feature channels;]. The input data 200 may be expressed in width, height and depth. Details of input data 210 with one channel are shown in terms of width, height and length. In FIGS. 2A and 2B, 210 to 240 show an example of data processed by a convolution operation and pooling operation with respect to data inputted to one channel. Data inputted to other channels may be processed using the same method [the first group of pooled feature channels generated by the first window pooling process with the second group of pooled feature channels]. Input data 200 may be zero-padded in order to comply with the size of a convolution window. Zero padding may mean that 0 is inserted into the edge of the input data 210.”)
concatenating the merged pooled feature channels with the input feature channels to generate concatenated feature channels; and (Jin, paragraph 0046, “Referring to FIG. 2A, input data 200 on the leftmost side may include a plurality of channels. FIG. 2 shows an example in which the input data 200 includes three channels. The input data 200 may be expressed in width, height and depth. Details of input data 210 with one channel are shown in terms of width, height and length. In FIGS. 2A and 2B, 210 to 240 show an example of data processed by a convolution operation and pooling operation with respect to data inputted to one channel. Data inputted to other channels may be processed using the same method. Input data 200 may be zero-padded in order to comply with the size of a convolution window [input feature channels to generate concatenated feature channels]. Zero padding may mean that 0 is inserted into the edge of the input data 210.”)
applying a two-dimensional convolutional neural network (CNN) to the concatenated feature channels to generate contextually pooled output feature channels. (Jin, paragraph 0052, “Referring to FIG. 3, in an initialization operation 310, the CNN operation apparatus may receive and store pieces of information for executing a CNN algorithm. The pieces of received information may become weight data. Furthermore, the CNN operation apparatus may further receive and store at least one of layer configuration information, an address map for convolution input (or a precomputed memory map) and an activation function look-up table (LUT). After performing the initialization operation 310, the CNN operation apparatus may perform a feature extraction operation [a two-dimensional convolutional neural network (CNN) to the concatenated feature channels] 320. The feature extraction operation 320 may become a convolution operation and a pooling operation. The convolution operation and the pooling operation may be performed according to a method and procedure, such as those shown in FIGS. 1 to 2B. When completing the feature extraction operation 320 for the input data, the CNN operation apparatus may perform a classification operation 330. The classification operation 330 may include at least one fully connected layer as shown in FIG. 1. When completing the classification operation 330, the CNN operation apparatus may output data to the outside of a system in a finish operation [generate contextually pooled output feature channels. ] 340.”)
However, Jin does not explicitly teach segmenting the plurality of input feature channels into a first group of feature channels and a second group of feature channels;
Avital teaches segmenting the plurality of input feature channels into a first group of feature channels and a second group of feature channels; (Avital, paragraph 0158 and paragraph 0159, “The number of feature channels doubles for each sequential stage of the encoding-contracting path, and is halved for each sequential stage of the decoding-expanding path [plurality of input feature channels into a first group of feature channels and a second group of feature channels;]. For example, the number of 3D voxel filters (each corresponding to a respective feature channel) at each stage of each instance of the common component are: I 0, 20, 40, 80, and 160.”)
Jin and Avital are related to the same field of endeavor (i.e. convolutional neural networks). In view of the teachings of Avital, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Avital to Jin before the effective filing date of the claimed invention in order to improve the ability to automatically segment predefined regions. (Avital, paragraph 0014, “The systems, apparatus, methods and/or code instructions (stored in a data storage device, executable by one or more hardware processors) described herein are directed to an improvement in computer-related technology, by improving the ability of computers to automatically segment predefined anatomical regions based on multimodal images.”)
However, Jin and Avital do not explicitly teach receiving an input image comprising an array of pixels, at least a portion of the pixels having assigned thereto a pixel value;
Extracting a plurality of input feature channels from the input image, wherein each input feature channel is a sub-array of pixels extracted from the input image;
wherein the first windowed pooling process applies a first operation to a plurality of horizontally adjacent pixels in each feature channel in the first group of feature channels,
wherein the second windowed pooling process applies a second operation to a plurality of vertically adjacent pixels in each feature channel in the second group of feature channels,
wherein each pooled feature channel in an aggregated group of the first group of pooled feature channels and the second group of pooled feature channels has been processed by exactly one of the first operation or the second operation.
Kim teaches receiving an input image comprising an array of pixels, at least a portion of the pixels having assigned thereto a pixel value; (Kim, paragraph 0004, Kim teaches generating feature maps from an input image. And paragraph 0019, Kim teaches using a processor to generate an output feature map. Each value in the feature map correspond to an output pixel, determined by applying a three-dimensional weight of K*K*N, as shown in paragraph 0048. Since the values in the feature map are associated with specific pixels, Kim teaches determining the pixel values.)
extracting a plurality of input feature channels from the input image, wherein each input feature channel is a sub-array of pixels extracted from the input image; (Kim, paragraph 0011, Kim teaches obtaining address information of the feature map and the plurality of input pixels contained in the input feature map. The foregoing teaches extracting a plurality of input feature channels from the input image.)
wherein the first windowed pooling process applies a first operation to a plurality of horizontally adjacent pixels in each feature channel in the first group of feature channels, (Kim, paragraph 0048, “That is, the value of the output pixel at a particular position in an output feature map is determined by applying a three-dimensional weight of K*K*N around the adjacent input pixels [adjacent pixels in each feature channel in the first group of feature channels] at the corresponding positions of the N input feature maps, the input feature map is multiplied by the values of the input pixels, added together, and then added together with the bias corresponding to the output feature map.” and paragraph 0112, “The code below represents a method of generating an address of a scheme including steps of processing the coordinates of the output feature map vertically and horizontally, processing pooling positions [first windowed pooling process applies a first operation] of vertical and horizontal directions [a plurality of horizontally adjacent pixels] in itself, processing K*K weights for each value, and processing in a channel direction initially for each weight position (in the manner of processing for each of K*K by direction of N).”)
wherein the second windowed pooling process applies a second operation to a plurality of vertically adjacent pixels in each feature channel in the second group of feature channels, (Kim, paragraph 0048, “That is, the value of the output pixel at a particular position in an output feature map is determined by applying a three-dimensional weight of K*K*N around the adjacent input pixels [adjacent pixels in each feature channel in the second group of feature channels] at the corresponding positions of the N input feature maps, the input feature map is multiplied by the values of the input pixels, added together, and then added together with the bias corresponding to the output feature map.” and paragraph 0112, “The code below represents a method of generating an address of a scheme including steps of processing the coordinates of the output feature map vertically and horizontally, processing pooling positions [the second windowed pooling process applies a second operation] of vertical [a plurality of vertically adjacent pixels] and horizontal directions in itself, processing K*K weights for each value, and processing in a channel direction initially for each weight position (in the manner of processing for each of K*K by direction of N).”)
wherein each pooled feature channel in an aggregated group of the first group of pooled feature channels and the second group of pooled feature channels has been processed by exactly one of the first operation or the second operation; (Kim, paragraph 0048, “That is, the value of the output pixel at a particular position in an output feature map is determined by applying a three-dimensional weight of K*K*N around the adjacent input pixels at the corresponding positions of the N input feature maps, the input feature map is multiplied by the values of the input pixels, added together, and then added together with the bias corresponding to the output feature map [the first group of pooled feature channels and the second group of pooled feature channels, examiner interprets the input feature map as the first group of pooled feature channels and then the output feature map is interpreted as the second group of feature channels.].” and paragraph 0049, In addition, the CNN processor may perform pooling after such convolution and activation, for example, by selecting the largest value for a given window size, for example, a 2*2 window, or by reducing the size of the feature map [has been processed by exactly one of the first operation]. Depending on the implementation, convolution, batch normalization, activation, and pooling may be called individual layers, or a combination of several thereof may be defined as one layer.” And paragraph 0112, “The code below represents a method of generating an address of a scheme including steps of processing the coordinates of the output feature map vertically and horizontally, processing pooling positions of vertical and horizontal directions in itself, processing K*K weights for each value, and processing in a channel direction initially for each weight position (in the manner of processing for each of K*K by direction of N).”)
Jin, Avital and Kim are related to the same field of endeavor (i.e. convolutional neural networks). In view of the teachings of Kim, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Kim to Jin and Avital before the effective filing date of the claimed invention in order to efficiently store the input and output feature maps. (Kim, paragraph 0008, “Embodiments of the present invention provide an apparatus for processing a convolutional neural network using a systolic array and a method thereof using the operational result for one layer as an input to the operation for a next layer, while using the systolic array easily, and efficiently storing an input feature map and an output feature map.”)
In regard to claim 17, Jin, Avital and Kim teach the product of claim 14.
Jin further teaches wherein the merging is a weighted merging performed with weighting factors generated by a gate selection CNN. (Jin, paragraph 0046, “Referring to FIG. 2A, input data 200 on the leftmost side may include a plurality of channels. FIG. 2 shows an example in which the input data 200 includes three channels. The input data 200 may be expressed in width, height and depth. Details of input data 210 with one channel are shown in terms of width, height and length. In FIGS. 2A and 2B, 210 to 240 show an example of data processed by a convolution operation and pooling operation with respect to data inputted to one channel. Data inputted to other channels may be processed using the same method. Input data 200 may be zero-padded in order to comply with the size of a convolution window. Zero padding may mean that 0 is inserted into the edge of the input data 210.” And paragraph 0122, “An OR gate 820 may perform OR operation on a Z_F, included in an input data set having the same architecture as the input data set 911, and a Z_F Included in a weight data set having the same architecture as the weight data set 913 [weighting factors generated by a gate selection CNN]. A selector (or multiplexer) 830 may first input the output of the multiplier 810 (i.e., the first input), may second input a ground (or zero) signal (i.e., the second input), may select and out one of the first input or the second input based on the output of the OR gate 820. Each of the PEs of the PE unit 545 may include the multiplier 810 corresponding to the number of points of input data. For example, each PE may be a 16-bit or 32-bit/floating or fixed point multiplier. Each PE may receive an input data set and a weight data set, such as those shown in FIG. 9A, under the control of the PE controller 543. The multiplier 810 may perform convolution multiplication operation on the received two data.”)
Claims 3 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Jin, in view of Avital, Kim and in further view of Scherer et al (Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition, "Scherer").
In regard to claim 3 and analogous claim and 16, Jin, Avital and Kim teach the method of claim 1.
However, Jin, Avital and KIm do not explicitly teach wherein the first windowed pooling process employs a first pooling kernel of length greater than 16 and the second windowed pooling process employs a second pooling kernel of length greater than 16.
Scherer teaches wherein the first windowed pooling process employs a first pooling kernel of length greater than 16 and the second windowed pooling process employs a second pooling kernel of length greater than 16. (Scherer, pg. 97, 4.2 Max Pooling versus Subsampling, paragraph 4, “For Caltech-101, the input layer consists of three feature maps of size 140×140, followed by a convolutional layer C1 with 16 × 16 filters and 16 feature maps. The subsequent pooling layer P2 reduces the 125 × 125 maps with 5 × 5 nonoverlapping pooling windows to a size of 25 × 25 pixels.”)
Jin, Avital, Kim and Scherer are related to the same field of endeavor (i.e. convolutional neural networks). In view of the teachings of Scherer, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Scherer to Jin, Avital and Kim before the effective filing date of the claimed invention in order to have low error rates for the process. (Scherer, Abstract, “By applying this knowledge, we achieve state-of-the-art error rates of 4.57% on the NORB normalized-uniform dataset and 5.6% on the NORB jittered-cluttered dataset.”)
Claim 7-9, 11-13 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Jin, in view of Avital, Kim and in further view of Ling et al (US Published Patent Application No. 20200160111, "Ling").
In regard to claim 7 and analogous claims 13 and 20, Jin, Avital and Kim teach the method of claim 1.
Jin further teaches applying a backbone CNN to the input image to extract the plurality of input feature channels from the input image; and. (Jin, paragraph 0149, “As shown in FIG. 3, the CNN operation apparatus in accordance with various embodiments of the present invention may perform a feature extraction layer operation 320 on the input data [from the input image] of all of channels after performing an initialization operation 310. [a backbone CNN to an input image to extract the plurality input feature channels;] When the feature extraction layer operation 320 is completed, the CNN operation apparatus may perform a classification layer operation 330. When the classification layer operation 330 is completed, the CNN operation apparatus may send a learnt result value to the host.”)
However, Jin, Avital and Kim do not explicitly teach applying the contextually pooled output feature channels to an output CNN to generate a detection and/or a class prediction for one or more objects in the input image.
Ling teaches applying the contextually pooled output feature channels to an output CNN to generate a detection and/or a class prediction for one or more objects in the input image. (Ling, paragraph 0059, “applying a regression convolutional neural network (CNN) to an image, the regression CNN to predict properties of a copy space, the properties including size and type, the prediction conditioned on a determination by the CNN of a copy space presence in the image [class prediction]; and applying a segmentation CNN to the image, the segmentation CNN to generate one or more masks associated with locations of one or more of a manufactured copy space in the image, a natural copy space in the image, and a background region of the image.”)
Jin, Avital, Kim and Ling are related to the same field of endeavor (i.e. convolutional neural networks). In view of the teachings of Ling, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Ling to Jin, Avital and Kim before the effective filing date of the claimed invention in order to improve workflow efficiency. (Ling, paragraph 0015, “The disclosed techniques improve workflow efficiency, reduce production time, and lower the cost of copy space based image production.”)
In regard to claim 8, Jin teaches and wherein applying the first windowed pooling process generates a first group of pooled feature channels; (Jin, paragraph 0043, “The feature extraction part 110 may include an architecture in which output data processed by a pair of a convolution layer and a pooling layer becomes the input data of a next pair of a convolution layer and a pooling layer. For example, external input data 100 may become the input data of the first convolution layer of the feature extraction part 110, and data processed and output by the first convolution layer may become the input data of the first pooling layer.” Also see 0046 and 0048)
apply a second windowed pooling process to the second group of feature channels, (Jin, paragraph 0043, Jin teaches the second window pooling process on the second group of feature channels. Jin discusses the pooling process being performed as many times as set. Allowing for the second pooling process to be done on the second group of feature channels.)
wherein applying the second windowed pooling process generates a second group of pooled feature channels, (Jin, paragraph 0043, “Furthermore, data processed by the first pooling layer [generates a second group of pooled feature channels, examiner interprets the pooling layer as the second pooled feature channel due to the pooling operation being able to perform as many times as necessary]] may become the input data of a second convolution layer, that is, one of a next pair. A convolution operation and a pooling operation may be performed by the number of times [wherein applying the second windowed pooling] (e.g., N times) set in a host.” Also see 0046 and 0048)
perform a merging of the first group of pooled feature channels and the second group of pooled feature channels to generate merged pooled feature channels; (Jin, paragraph 0046, “Referring to FIG. 2A, input data 200 on the leftmost side may include a plurality of channels. FIG. 2 shows an example in which the input data 200 includes three channels [generate merged pooled feature channels;]. The input data 200 may be expressed in width, height and depth. Details of input data 210 with one channel are shown in terms of width, height and length. In FIGS. 2A and 2B, 210 to 240 show an example of data processed by a convolution operation and pooling operation with respect to data inputted to one channel. Data inputted to other channels may be processed using the same method [the first group of pooled feature channels generated by the first window pooling process with the second group of pooled feature channels]. Input data 200 may be zero-padded in order to comply with the size of a convolution window. Zero padding may mean that 0 is inserted into the edge of the input data 210.”)
concatenate the merged pooled feature channels with the plurality of input feature channels to generate concatenated feature channels; (Jin, paragraph 0046, “Referring to FIG. 2A, input data 200 on the leftmost side may include a plurality of channels. FIG. 2 shows an example in which the input data 200 includes three channels. The input data 200 may be expressed in width, height and depth. Details of input data 210 with one channel are shown in terms of width, height and length. In FIGS. 2A and 2B, 210 to 240 show an example of data processed by a convolution operation and pooling operation with respect to data inputted to one channel. Data inputted to other channels may be processed using the same method. Input data 200 may be zero-padded in order to comply with the size of a convolution window [input feature channels to generate concatenated feature channels]. Zero padding may mean that 0 is inserted into the edge of the input data 210.”)
wherein the second windowed pooling process applies a mean pooling process to a plurality of vertically adjacent pixels in each feature channel in the second group of feature channels, (Jin, paragraph 0048, “When the convolution operation is performed, output data, such as data 230 of FIG. 2B, may be generated. When the convolution operation is terminated, the output data may be computed as an activation value through one of activation functions 240. In FIG. 2B, "249" shows an example in which an activation function is expressed in equation. An activation output value may become the input data of the pooling layer 250. A pooling operation may be an operation for selecting a maximum value of the values of a plurality of input data that enters a pooling window, such as a polling window 241 or 242 in FIG. 2B, or calculating an average value of the plurality of input data [a mean pooling process]. Such maximum values or average values may be compressed into one value, such as 251 or 252. In FIG. 2B, "259" shows an example in which the pooling operation is expressed in equation.”)
apply a two-dimensional convolutional neural network (CNN) to the concatenated feature channels to generate contextually pooled output feature channels; and (Jin, paragraph 0052, “Referring to FIG. 3, in an initialization operation 310, the CNN operation apparatus may receive and store pieces of information for executing a CNN algorithm. The pieces of received information may become weight data. Furthermore, the CNN operation apparatus may further receive and store at least one of layer configuration information, an address map for convolution input (or a precomputed memory map) and an activation function look-up table (LUT). After performing the initialization operation 310, the CNN operation apparatus may perform a feature extraction operation [a two-dimensional convolutional neural network (CNN) to the concatenated feature channels] 320. The feature extraction operation 320 may become a convolution operation and a pooling operation. The convolution operation and the pooling operation may be performed according to a method and procedure, such as those shown in FIGS. 1 to 2B. When completing the feature extraction operation 320 for the input data, the CNN operation apparatus may perform a classification operation 330. The classification operation 330 may include at least one fully connected layer as shown in FIG. 1. When completing the classification operation 330, the CNN operation apparatus may output data to the outside of a system in a finish operation [generate contextually pooled output feature channels. ] 340.”)
However, Jin does not explicitly teach receive an input image comprising an array of pixels, at least a portion of the pixels having assigned thereto a pixel value;
extract a plurality of input feature channels from the input image;
segment the plurality of input feature channels into a first group of feature channels and a second group of feature channels;
apply a first windowed pooling process to a plurality of horizontally adjacent pixels in each feature channel in the first group of feature channels,
wherein each pooled feature channel in an aggregated group of the first group of pooled feature channels and the second group of pooled feature channels was generated by exactly one of the first window pooling process or the second window pooling process;
apply the contextually pooled output feature channels to an output CNN to generate a detection and/or a class prediction for one or more objects in the input image
Avital teaches segment the plurality of input feature channels into a first group of feature channels and a second group of feature channels; (Avital, paragraph 0158 and paragraph 0159, “The number of feature channels doubles for each sequential stage of the encoding-contracting path, and is halved for each sequential stage of the decoding-expanding path [plurality of input feature channels into a first group of feature channels and a second group of feature channels;]. For example, the number of 3D voxel filters (each corresponding to a respective feature channel) at each stage of each instance of the common component are: I 0, 20, 40, 80, and 160.”)
Jin and Avital are related to the same field of endeavor (i.e. convolutional neural networks). In view of the teachings of Avital, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Avital to Jin before the effective filing date of the claimed invention in order to improve the ability to automatically segment predefined regions. (Avital, paragraph 0014, “The systems, apparatus, methods and/or code instructions (stored in a data storage device, executable by one or more hardware processors) described herein are directed to an improvement in computer-related technology, by improving the ability of computers to automatically segment predefined anatomical regions based on multimodal images.”)
However, Jin and Avital do not explicitly teach receive an input image comprising an array of pixels, at least a portion of the pixels having assigned thereto a pixel value;
extract a plurality of input feature channels from the input image;
apply a first windowed pooling process to a plurality of horizontally adjacent pixels in each feature channel in the first group of feature channels,
wherein the second windowed pooling process applies a mean pooling process to a plurality of vertically adjacent pixels in each feature channel in the second group of feature channels,
wherein each pooled feature channel in an aggregated group of the first group of pooled feature channels and the second group of pooled feature channels was generated by exactly one of the first window pooling process or the second window pooling process;
apply the contextually pooled output feature channels to an output CNN to generate a detection and/or a class prediction for one or more objects in the input image.
Kim teaches receive an input image comprising an array of pixels, at least a portion of the pixels having assigned thereto a pixel value; (Kim, paragraph 0004, Kim teaches generating feature maps from an input image. And paragraph 0019, Kim teaches using a processor to generate an output feature map. Each value in the feature map correspond to an output pixel, determined by applying a three-dimensional weight of K*K*N, as shown in paragraph 0048. Since the values in the feature map are associated with specific pixels, Kim teaches determining the pixel values.)
extract a plurality of input feature channels from the input image; (Kim, paragraph 0011, Kim teaches obtaining address information of the feature map and the plurality of input pixels contained in the input feature map. The foregoing teaches extracting a plurality of input feature channels from the input image.)
apply a first windowed pooling process to a plurality of horizontally adjacent pixels in each feature channel in the first group of feature channels, (Kim, paragraph 0048, “That is, the value of the output pixel at a particular position in an output feature map is determined by applying a three-dimensional weight of K*K*N around the adjacent input pixels [adjacent pixels in each feature channel in the first group of feature channels] at the corresponding positions of the N input feature maps, the input feature map is multiplied by the values of the input pixels, added together, and then added together with the bias corresponding to the output feature map.” and paragraph 0112, “The code below represents a method of generating an address of a scheme including steps of processing the coordinates of the output feature map vertically and horizontally, processing pooling positions [first windowed pooling process applies a first operation] of vertical and horizontal directions [a plurality of horizontally adjacent pixels] in itself, processing K*K weights for each value, and processing in a channel direction initially for each weight position (in the manner of processing for each of K*K by direction of N).”)
wherein each pooled feature channel in an aggregated group of the first group of pooled feature channels and the second group of pooled feature channels was generated by exactly one of the first window pooling process or the second window pooling process; (Kim, paragraph 0048, “That is, the value of the output pixel at a particular position in an output feature map is determined by applying a three-dimensional weight of K*K*N around the adjacent input pixels at the corresponding positions of the N input feature maps, the input feature map is multiplied by the values of the input pixels, added together, and then added together with the bias corresponding to the output feature map [the first group of pooled feature channels and the second group of pooled feature channels, examiner interprets the input feature map as the first group of pooled feature channels and then the output feature map is interpreted as the second group of feature channels.].” and paragraph 0049, In addition, the CNN processor may perform pooling after such convolution and activation, for example, by selecting the largest value for a given window size, for example, a 2*2 window, or by reducing the size of the feature map [has been processed by exactly one of the first operation]. Depending on the implementation, convolution, batch normalization, activation, and pooling may be called individual layers, or a combination of several thereof may be defined as one layer.” And paragraph 0112, “The code below represents a method of generating an address of a scheme including steps of processing the coordinates of the output feature map vertically and horizontally, processing pooling positions of vertical and horizontal directions in itself, processing K*K weights for each value, and processing in a channel direction initially for each weight position (in the manner of processing for each of K*K by direction of N).”)
Jin, Avital and Kim are related to the same field of endeavor (i.e. convolutional neural networks). In view of the teachings of Kim, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Kim to Jin and Avital before the effective filing date of the claimed invention in order to efficiently store the input and output feature maps. (Kim, paragraph 0008, “Embodiments of the present invention provide an apparatus for processing a convolutional neural network using a systolic array and a method thereof using the operational result for one layer as an input to the operation for a next layer, while using the systolic array easily, and efficiently storing an input feature map and an output feature map.”)
However, Jin, Avital and Kim do not explicitly teach apply the contextually pooled output feature channels to an output CNN to generate a detection and/or a class prediction for one or more objects in the input image.
Ling teaches apply the contextually pooled output feature channels to an output CNN to generate a detection and/or a class prediction for one or more objects in the input image. (Ling, paragraph 0059, “applying a regression convolutional neural network (CNN) to an image, the regression CNN to predict properties of a copy space, the properties including size and type, the prediction conditioned on a determination by the CNN of a copy space presence in the image [class prediction]; and applying a segmentation CNN to the image, the segmentation CNN to generate one or more masks associated with locations of one or more of a manufactured copy space in the image, a natural copy space in the image, and a background region of the image.”)
Jin, Avital, Kim and Ling are combinable for the same rationale as set forth above with respect to claim 7.
In regard to claim 9, Jin, Avital, Kim and Ling teach the system of claim 8.
Jin further teaches wherein the first windowed pooling process is a maximum pooling process or a minimum pooling process. (Jin, paragraph 0048, “When the convolution operation is performed, output data, such as data 230 of FIG. 2B, may be generated. When the convolution operation is terminated, the output data may be computed as an activation value through one of activation functions 240. In FIG. 2B, "249" shows an example in which an activation function is expressed in equation. An activation output value may become the input data of the pooling layer 250. A pooling operation may be an operation for selecting a maximum value of the values of a plurality of input data that enters a pooling window [maximum pooling process], such as a polling window 241 or 242 in FIG. 2B, or calculating an average value of the plurality of input data. Such maximum values or average values may be compressed into one value, such as 251 or 252. In FIG. 2B, "259" shows an example in which the pooling operation is expressed in equation.”)
In regard to claim 11, Jin, Avital, Kim and Ling teach the system of claim 8.
Jin further teaches wherein the merging is a weighted merging performed with weighting factors generated by a gate selection CNN. (Jin, paragraph 0046, “Referring to FIG. 2A, input data 200 on the leftmost side may include a plurality of channels. FIG. 2 shows an example in which the input data 200 includes three channels. The input data 200 may be expressed in width, height and depth. Details of input data 210 with one channel are shown in terms of width, height and length. In FIGS. 2A and 2B, 210 to 240 show an example of data processed by a convolution operation and pooling operation with respect to data inputted to one channel. Data inputted to other channels may be processed using the same method. Input data 200 may be zero-padded in order to comply with the size of a convolution window. Zero padding may mean that 0 is inserted into the edge of the input data 210.” And paragraph 0122, “An OR gate 820 may perform OR operation on a Z_F, included in an input data set having the same architecture as the input data set 911, and a Z_F Included in a weight data set having the same architecture as the weight data set 913 [weighting factors generated by a gate selection CNN]. A selector (or multiplexer) 830 may first input the output of the multiplier 810 (i.e., the first input), may second input a ground (or zero) signal (i.e., the second input), may select and out one of the first input or the second input based on the output of the OR gate 820. Each of the PEs of the PE unit 545 may include the multiplier 810 corresponding to the number of points of input data. For example, each PE may be a 16-bit or 32-bit/floating or fixed point multiplier. Each PE may receive an input data set and a weight data set, such as those shown in FIG. 9A, under the control of the PE controller 543. The multiplier 810 may perform convolution multiplication operation on the received two data.”)
In regard to claim 12, Jin, Avital, Kim and Ling teach the system of claim 8.
Ling further teaches wherein one or more processors are further configured to apply the contextually pooled output feature channels to an output CNN to generate a detection and/or a class prediction for one or more objects in the input image. (Ling, paragraph 0059, “applying a regression convolutional neural network (CNN) to an image, the regression CNN to predict properties of a copy space, the properties including size and type, the prediction conditioned on a determination by the CNN of a copy space presence in the image [class prediction]; and applying a segmentation CNN to the image, the segmentation CNN to generate one or more masks associated with locations of one or more of a manufactured copy space in the image, a natural copy space in the image, and a background region of the image.”)
Jin, Avital, Kim and Ling are combinable for the same rationale as set forth above with respect to claim 7.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Jin, in view of Avital, Kim and Ling, in further view of Scherer.
In regard to claim 10, Jin, Avital, Kim and Ling teach the system of claim 8.
However, Jin, Avital, Kim and Ling do not explicitly teach wherein the first windowed pooling process employs a first pooling kernel of length greater than 16 and the second windowed pooling process employs a second pooling kernel of length greater than 16.
Scherer teaches wherein the first windowed pooling process employs a first pooling kernel of length greater than 16 and the second windowed pooling process employs a second pooling kernel of length greater than 16. (Scherer, pg. 97, 4.2 Max Pooling versus Subsampling, paragraph 4, “For Caltech-101, the input layer consists of three feature maps of size 140×140, followed by a convolutional layer C1 with 16 × 16 filters and 16 feature maps. The subsequent pooling layer P2 reduces the 125 × 125 maps with 5 × 5 nonoverlapping pooling windows to a size of 25 × 25 pixels.”)
Jin, Avital, Kim, Ling and Scherer are related to the same field of endeavor (i.e. convolutional neural networks). In view of the teachings of Scherer, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Scherer to Jin, Avital, Kim and Ling before the effective filing date of the claimed invention in order to have low error rates for the process. (Scherer, Abstract, “By applying this knowledge, we achieve state-of-the-art error rates of 4.57% on the NORB normalized-uniform dataset and 5.6% on the NORB jittered-cluttered dataset.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SKYLAR K VANWORMER whose telephone number is (703)756-1571. The examiner can normally be reached M-F 6:00am to 3:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached on (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.K.V./Examiner, Art Unit 2146 /USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146