DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s response to the last Office Action, filed 4/28/2026, has been entered and made of record.
Applicant has amended claims 1,6,9,10,20. Claims 21-22 have been added. Claims 1-22 are currently pending.
Applicants arguments filed 4/28/2026 have been fully considered but they are not persuasive.
Applicant argues that Dekel discloses that a "skip architecture is defined, that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations." Dekel at para. [0125]. Dekel, however, does not disclose or suggest that the skip architecture is "based on element-wise addition" or that a "U-shaped network" uses the skip architecture. Therefore, Dekel does not disclose or suggest "wherein an encoder of the U-shaped network contributes to a decoder of the U-shaped network using a skip connection based on element-wise addition," as recited by claim 1. Additionally the applicant argues that Ikonin does not teach the newly added limitation "a first stage comprising a pooling input mixer followed by a first channel scaler, and a second stage comprising a multi-layer perceptron followed by a second channel scaler," as recited by claim 1.
In response, Dekel teaches a skip architecture is defined, that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations (paragraph [0125]),such an architecture is characteristic of a U shaped encoder that is also taught be Dekel (paragraph [0124])in which feature maps from encoded layers are transmitted to corresponding decoder layers through a skip connection. Dekel teaches “Using nearest neighbor up sampling, the last feature map from the bottom-up pathway is expanded to the same scale as the second-to-last feature map. These two feature maps are then merged by element-wise addition to form a new feature map.” ( paragraph [0345]) which shows that the skip connection can employ element wise addition where a person of ordinary skill in the art would have recognized that the disclosed transfer of information between deeper and shallower network layers is implemented through conventional skip connections used in encoder decoder segmentations architectures. At the time U-NET and related U-shaped architecture commonly employed skip connections to convey encoder features to decoder stages using concatenation ( paragraph [0124]) or element wise addition ( paragraph [0108]). Therefore Dekels disclosure of combining information from deep and shallow layers through a skip architecture reasonably suggests a U Shaped encoder decoder arrangement having skip connection between corresponding encoder and decoder layers. Selection of element wise addition as the merging mechanism would have been obvious design choice among a finite number of well know alternatives for implementing the disclosed skip architecture yielding predictable results. Examiner used a secondary reference Ikonin that teaches a neural network block the pooling layer (paragraph [0099-0102] followed by MLP stage where the average pooling filter shape and scaling factor for the upsampling operation ( paragraph [0250]). In view of the new ground(s) of rejection necessitated by the amendments, examiner will use a different reference that teaches the newly added limitation that the transformer block comprises two stages and does not comprise a self-attention function. All remaining arguments are reliant on the aforementioned and addressed arguments and thus are considered to be wholly addressed herein.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Dekel et al (US 2024/0419382) in view of Tolstikhin et al (MLP-Mixer: An all-MLP Architecture for Vision)
As to claim 1, Dekel et al teaches an apparatus for image restoration, the apparatus comprising: one or more memories storing instructions; and one or more processors configured to execute the instructions to at least:
operate on an input image ( operate generator, 63; figure 6 with image 62a,62b) using a U-shaped network ( u-shaped network, paragraph [0124]) to produce an output image( generate output image 64, figure 6),
wherein an encoder of the U-shaped network contributes to a decoder of the U- shaped network(the encoder-decoder attention, paragraph [0202] note that each encoder consists of two major components: a self-attention mechanism and a feed-forward neural network. The self-attention mechanism accepts input encodings from the previous encoder and weighs their relevance to each other to generate output encodings. The feed-forward neural network further processes each output encoding individually. , paragraph [0201]) using a skip connection based on element-wise addition (A skip architecture is defined, that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations, paragraph [0125]),
the encoder is a first instance of a transformer block (the transformer Encoder 44 may include one or more layers, paragraph [0207] ; the original Transformer model architecture uses an encoder-decoder architecture, paragraph [019]) , the decoder is a second instance of the transformer block, and the transformer block comprises (Each of the layers may comprise 3×3 Convolutions, followed by BatchNorm, and LeakyReLU activation. Any ANN herein, may comprise, may use, or may be based on, a method, scheme or architecture such as U-Net, which is based on the fully convolutional network to supplement a usual contracting network by successive layers, where pooling operations are replaced by upsampling operators. These layers increase the resolution of the output, and a successive convolutional layer can then learn to assemble a precise output based on this information, paragraph [0340] ) . While Dekel meets the limitation above. Dekel fails to teach “ a first stage comprising a pooling input mixer followed by a first channel scaler, and a second stage comprising a multi-layer perceptron followed by a second channel scaler. “ Specifically, Tolstikhin et al teaches the token mixing MLPs that allow communication between different spatial locations(tokens) and then applies channel mixing MLPS ( per token feature scaling/interaction, second stage) .Tolstikhin teaches Figure 1: MLP-Mixer consists of per-patch linear embeddings, Mixer layers, and a classifier head. Layer Norm Channels Layer Norm Mixer layers contain one token-mixing MLP and one channel-mixing MLP, each consisting of two fully-connected layers and a GELU nonlinearity. Other components include: skip-connections, dropout, and layer norm on the channels. they operate on each token independently and take individual rows of the table as inputs. The token-mixing MLPs allow communication between different spatial locations (tokens); they operate on each channel independently and take individual columns of the table as inputs. These two types of layers are interleaved to enable interaction of both input dimensions.(Section 1 and 2) Therefore the feature wise transformation ( scaling per chanel) correspond to the first channel scaler and the second channel mixing stage after anather transformation correspond to the later channel mixing MLP in deeper layer .It would have been obvious to one skilled in the art before filing of the claimed invention to use the MLP-Mixer architecture as taught by Tolstikhin in order to enhance the tracking performance and improve ImageNet accuracy for the larger scales. Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
As to claim 2, Tolstikhin et al teaches the apparatus of claim 1, wherein the pooling input mixer doesn't comprise learnable parameters (Aside from the MLP layers, Mixer uses other standard architectural components: skip-connec tions [15] and layer normalization [2]. Unlike ViTs, Mixer does not use position embeddings because the token-mixing MLPs are sensitive to the order of the input tokens. Finally, Mixer uses a standard classification head with the global average pooling layer followed by a linear classifier. Overall, the architecture can be written compactly in JAX/Flax, the code is given in Supplementary ; section 2)
As to claim 3, Tolstikhin et al teaches the apparatus of claim 1, wherein an input shape of the pooling input mixer is equal to an output shape of the pooling input mixer ( input tokens and output tokens , mixer architecture section 2).
As to claim 4, Dekel teaches the apparatus of claim 1, wherein the U-shaped network does not apply concatenation to the skip connection thereby reducing latency of the apparatus by avoiding a dimensionality expansion The contracting path is a typical convolutional network that consists of repeated application of convolutions, each followed by a rectified linear unit (ReLU) and a max pooling operation. During the contraction, the spatial information is reduced while feature information is increased, paragraph [0124]).
As to claim 5, Dekel teaches the apparatus of claim 1, wherein the pooling input mixer is preceded in the transformer block by a batch normalization, and the batch normalization is configured to be foldable into a preceding linear transformation (Each of the layers may comprise 3×3 Convolutions, followed by BatchNorm, and LeakyReLU activation. The encoder's channels dimensions are [3.fwdarw.16.fwdarw.32.fwdarw.64.fwdarw.128.fwdarw.128], while the decoder follows a reversed order. In each level of the encoder, an additional 1×1 Convolution layer is added and concatenate the output features to the corresponding level of the decoder. Lastly, a 1×1 Convolution layer is added followed by Sigmoid activation to get the final RGB output. Any ANN herein, may comprise, may use, or may be based on, a method, scheme or architecture such as U-Net, which is based on the fully convolutional network to supplement a usual contracting network by successive layers, where pooling operations are replaced by upsampling operators. These layers increase the resolution of the output, and a successive convolutional layer can then learn to assemble a precise output based on this information, paragraph [0340]).
As to claim 6, Dekel teaches the apparatus of claim 1, wherein the transformer block comprises a first stage and a second stage, the first stage comprises a first batch normalization, the pooling input mixer, the first channel scaler, and a first summation node, wherein the first summation node operates on an input to the transformer block and an output of the first channel scaler, and the second stage comprises a second batch normalization, the multi-layer perceptron followed by the second channel scaler, and a second summation node, wherein the second summation node operates on an output of the first summation node and an output of the second channel scaler to produce an output of the transformer block( The generator 52 may be based on, may include, or may use, an Artificial Neural Network (ANN) model or architecture. In one example, A U-Net architecture, with a 5-layer encoder and a symmetrical decoder. Each of the layers may comprise 3×3 Convolutions, followed by BatchNorm, and LeakyReLU activation. Any ANN herein, may comprise, may use, or may be based on, a method, scheme or architecture such as U-Net, which is based on the fully convolutional network to supplement a usual contracting network by successive layers, where pooling operations are replaced by upsampling operators. These layers increase the resolution of the output, and a successive convolutional layer can then learn to assemble a precise output based on this information, paragraph[0340]).
As to claim 7, Dekel teaches the apparatus of claim 1, wherein a latency of the apparatus is reduced by achieving a batch normalization which is latency-favorable on GPU hardware, a memory footprint is reduced by avoiding concatenation at the skip connection, and the avoiding concatenation allows a computation of a layer output to have a reduced latency by avoiding an increase in array dimensions (an efficient network architecture and a set of two hyper-parameters in order to build very small, low latency models that can be easily matched to the design requirements for mobile and embedded vision applications, and describes the MobileNet architecture and two hyper-parameters width multiplier and resolution multiplier to define smaller and more efficient MobileNets, paragraph [0121-0123]).
As to claim 8, Dekel teaches the apparatus of claim 1, wherein a first decoder at a same level of the U-shaped network as a first encoder is identical to the first encoder other than learned parameters ((Like the first encoder, the first decoder takes positional information and embeddings of the output sequence as its input, rather than encodings. The transformer must not use the current or future output to predict an output, so the output sequence must be partially masked to prevent this reverse information flow. This allows for autoregressive text generation. For all attention heads, attention can't be placed on following tokens. The last decoder is followed by a final linear transformation and softmax layer, to produce the output probabilities over the vocabulary, paragraph[0202]).
As to claim 9, Dekel teaches the smartphone ( Any apparatus herein, which may be any of the systems, devices, modules, or functionalities described herein, may be integrated with a smartphone, paragraph [0389])comprising: a camera app; a user interface; and an application specific integrated circuit, wherein the application specific integrated circuit ( an Application Programming Interface (API), defined as an intermediary software serving as the interface allowing the interaction and data sharing between an application software and the application platform, across which few or all services are provided, and commonly used to expose or use a specific software functionality, while protecting the rest of the application, paragraph [0390])is configured to at least: receive an input image from the camera app( operate generator, 63; figure 6 with image 62a,62b) , operate on the input image using a U-shaped network(u-shape network [0124]) to produce an output image(generate output image 64, figure 6), wherein an encoder of the U-shaped network contributes to a decoder of the U-shaped network (the encoder-decoder attention, paragraph [0202] note that each encoder consists of two major components: a self-attention mechanism and a feed-forward neural network. The self-attention mechanism accepts input encodings from the previous encoder and weighs their relevance to each other to generate output encodings. The feed-forward neural network further processes each output encoding individually. , paragraph [0201]) using a skip connection based on element-wise addition(a skip architecture is defined, that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations, paragraph [0125]), wherein the encoder is a first instance of a transformer (The Transformer Encoder 44 may include one or more layers, paragraph [0207] ; the original Transformer model architecture uses an encoder-decoder architecture, paragraph [019]) , the decoder is a second instance of the transformer (Each of the layers may comprise 3×3 Convolutions, followed by BatchNorm, and LeakyReLU activation. Any ANN herein, may comprise, may use, or may be based on, a method, scheme or architecture such as U-Net, which is based on the fully convolutional network to supplement a usual contracting network by successive layers, where pooling operations are replaced by upsampling operators. These layers increase the resolution of the output, and a successive convolutional layer can then learn to assemble a precise output based on this information, paragraph [0340] ) and transmit the output image to the camera app for display on the user interface (A sub flow chart 85c is executed by a separate device, that may be a server, such as the server 24, that received the first image as part of the “Receive Image” step 81a and the second image as part of the “Receive Image” step 82a. The received images are processed by a processor that executes the software or firmware of the generator 52 as part of the “Train/Operate Generator” step 63 for generating the output image as part of the “Generate Output Image” step 64. The produced output image 53 is sent, as part of a “Send Image” step 83 to a device that includes the display 54, where it is received as part of a “Receive Image” step 83a for displaying on the display 54. The device that houses the display 54 executes a sub flow chart 85d, that includes receiving of the output image 53 as part of the “Receive Image” step 83a, and displaying the received output image 53 on the display 54 as part of the “Display Output Image” step 65, paragraph [0364]). While Dekel meets the limitation above. Dekel fails to teach “ the transformer comprises a pooling input mixer followed by a first channel scaler, and a multi-layer perceptron followed by a second channel scaler. “ Specifically, Tolstikhin et al teaches the token mixing MLPs that allow communication between different spatial locations(tokens) and then applies channel mixing MLPS ( per token feature scaling/interaction, second stage) .Tolstikhin teaches Figure 1: MLP-Mixer consists of per-patch linear embeddings, Mixer layers, and a classifier head. Layer Norm Channels Layer Norm Mixer layers contain one token-mixing MLP and one channel-mixing MLP, each consisting of two fully-connected layers and a GELU nonlinearity. Other components include: skip-connections, dropout, and layer norm on the channels. they operate on each token independently and take individual rows of the table as inputs. The token-mixing MLPs allow communication between different spatial locations (tokens); they operate on each channel independently and take individual columns of the table as inputs. These two types of layers are interleaved to enable interaction of both input dimensions.(Section 1 and 2) Therefore the feature wise transformation ( scaling per chanel) correspond to the first channel scaler and the second channel mixing stage after anather transformation correspond to the later channel mixing MLP in deeper layer .It would have been obvious to one skilled in the art before filing of the claimed invention to use the MLP-Mixer architecture as taught by Tolstikhin in order to enhance the tracking performance and improve ImageNet accuracy for the larger scales. Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
The limitation of claims 10-20 has been addressed in claims 1-8 above.
As to claim 21, Tolstikhin teaches the apparatus of claim 1, wherein the transformer block does not comprise a self-attention function( M LP-Mixer is a new architecture for computer vision that differs from previous successful architectures because it uses neither convolutional nor self-attention layers, section 4).
As to claim 22, Tolstikhin teaches the apparatus of claim 6, wherein an output of the second batch normalization is based on the output of the first summation node (Mixer uses other standard architectural components: skip-connections [15] and layer normalization [2]; section 2)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NANCY BITAR whose telephone number is (571)270-1041. The examiner can normally be reached Mon-Friday from 8:00 am to 5:00 p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mrs. Jennifer Mehmood can be reached at 571-272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
NANCY . BITAR
Examiner
Art Unit 2664
/NANCY BITAR/Primary Examiner, Art Unit 2664
Prosecution Insights