Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments/Amendments
The amendment, filed 10/24/2025 in response to the Non-Final Office Action mailed on 07/24/2025 has been entered.
Claims 1-25 are currently pending in U.S. Patent Application No. 17/937,328.
The drawing objection made in the Non-Final Rejection is removed in view of the newly submitted Fig. 5.
The 35 U.S.C. 101 rejection made in the Non-Final Rejection is removed in view of the amended claims 7-12.
Regarding the 35 U.S.C. 103 rejections, the Applicant’s remarks have been fully considered but are moot because the new grounds of rejection regarding the amended limitation no longer relies on the combination of references presented in the Non-Final Rejection. A change in scope necessitated by the Applicant’s amendments has led to an updated search revealing new art. However, the Examiner notes below that while a new grounds of rejection is present, the Applicant remarks regarding the 35 U.S.C. 103 rejection are not persuasive for the following reasons. The Applicant assert’s that the combination made in the non-final office action does not at least describe “convert a plurality of multi-channel time synchronized signals into a plurality of images patches” or “combine the plurality of image patches into an image.”, citing that Naz works on single patient ECG signals and does not work with “multi-channel time-synchronized signals”. The Examiner respectfully disagrees. Naz discloses using a variety of different data sources for ECG data (see Table 1), wherein specifically the MIT-BIH dataset consists of two-lead ECG signals (i.e., “multi-channel”). The signals are “time-synchronized” such that the data obtained from both leads are obtained from the same starting time point (i.e., both leads obtain data starting at t=0, rather than one lead obtaining data at t=0 seconds and the other lead obtaining data at t=5 seconds). The Examiner notes the Moody and Mark paper which is referenced in the Naz reference (“The Impact of the MIT-BIH Arrhythmia Database”, DOI: 10.1109/51.932724), wherein Fig. 1 (shown below) from the Moody and Mark paper shows a sample ECG signal from the MIT-BIH dataset, which discloses signals from two-leads which are time-synchronized as they are plotted on the same x-axis.
PNG
media_image1.png
260
948
media_image1.png
Greyscale
PNG
media_image2.png
351
1052
media_image2.png
Greyscale
Furthermore, Naz provides the code for the method implementation (see the supplemental information provided by Naz), wherein the Matlab source file main.m (also attached to this Final Rejection) provides the logic used to process the multi-channel input data. The Examiner specifically notes the line X=reshape(T.val, [], 1) which involves reshaping the input data from multi-dimensional data (i.e., multi-channel time synchronized signals) into single-dimension data. Through this process, Naz takes multi-dimensional data and then further divides the data into image patches to generate a 32x32 image of the ECG signal.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5, 7-8, 11, 13-14, 17, 20-21, and 24 are rejected as being unpatentable over Li et al. (“Deep learning for digitizing highly noisy paper-based ECG records”, DOI: https://doi.org/10.1016/j.compbiomed.2020.104077; hereinafter “Li”) in view of Dosovitskiy et al. (“An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, DOI: arXiv:2010.11929v2; hereinafter “Dosovitskiy”).
Regarding Claim 1, Li discloses teaches a computing system comprising: a network controller; a processor coupled to the network controller; and a memory coupled to the processor, the memory including a set of instructions, which when executed by the processor, cause the processor to (Proposed Methodology, Results and Analysis, The Examiner notes that Li’s method involves complex signal and image processing using deep learning techniques, which are well understood to be performed on a computer machine. Consequently, the Examiner asserts that the computer machines used to perform the teachings of Li include the claimed “network controller”, “memory” and “processor”.): convert a plurality of multi-channel time-synchronized signals into a plurality of image patches (Fig. 3, 2.3. Proposed framework for ECG digitization, Li discloses processing a multi-lead ECG image (consisting of time-synchronized signals from multiple leads) by taking several 128x128 patches over the entire image.), combine the plurality of image patches into an image (Fig. 3, 2.3. Proposed framework for ECG digitization, Li discloses combining the patches of the multi-lead ECG image to generate a final image. The Examiner further notes the graphical abstract and supplemental figures provided by Li, which show how the image is processed in patches and then combined at the end to generate an overall ECG image.),
Li does not disclose generate, by an attention-based neural network, a classification result based on the image
Dosovitskiy teaches and generate, by an attention-based neural network, a classification result based on the image (Fig. 1, 3.1. Vision Transformer (ViT), Dosovitskiy teaches a Vision Transformer (ViT) network, which inputs an image through a transformer neural network and MLP head to produce a classification result. The Examiner notes that the ViT network disclosed by Dosovitskiy includes a multi-head attention module (see Fig. 1).).
Li and Dosovitskiy are considered to be analogous to the claimed invention as they are in the same field of using neural network models to analyze image data. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Li such that the image generated by Li is input into the ViT taught by Dosovitskiy in order to generate a classification result regarding the input image. The motivation for this combination being the ability to use a neural network model to classify the image.
Claims 7, 13, and 20 are the computer-readable storage medium, semiconductor apparatus, and method claims, respectively, corresponding to claim 1, and are similarly rejected (The Examiner notes that both Li and Dosovitskiy disclose methods for processing images which are commonly performed on computer machines. The Examiner asserts that the computer machines used by Li and Dosovitskiy include the claimed “computer-readable storage medium…executed by a computing system” and “semiconductor apparatus comprising one or more substrates”).
Regarding Claim 2, Li in view of Dosovitskiy teaches the computing system of claim 1, wherein the plurality of multi-channel time-synchronized signals are converted from a medical domain into the plurality of image patches, and wherein the plurality of multi-channel time-synchronized signals are to include one or more of electrocardiogram signals, electroencephalogram signals, electromyography signals or cardiotocography signals (2.1. Data Collection, Li discloses utilizing multi-lead ECG signal data.).
Claims 8, 14, and 21 are the computer-readable storage medium, semiconductor apparatus, and method claims, respectively, corresponding to claim 2, and are similarly rejected.
Regarding Claim 5, Li in view of Dosovitskiy teaches the computing system of claim 1, wherein the attention-based neural network is a two-dimensional transformer neural network (Fig. 1, 3.1. Vision Transformer (ViT), Dosovitskiy teaches a Vision Transformer (ViT) network which takes input of a two-dimensional image. The Examiner notes the similarities in network structure between Dosovitskiy’s Fig. 1 and the Applicant’s Fig. 5 as evidence to how Dosovitskiy’s ViT network is analogous to the claimed “two-dimensional transformer neural network”.).
Claims 11, 17, and 24 are the computer-readable storage medium, semiconductor apparatus, and method claims, respectively, corresponding to claim 5, and are similarly rejected.
Claims 3, 9, 15, and 22 are rejected as being unpatentable over Li in view of Dosovitskiy in view of Le et al. (“Multi-module Recurrent Convolutional Neural Network with Transformer Encoder for ECG Arrhythmia Classification”, DOI: 10.1109/BHI50953.2021.9508527; hereinafter “Le”).
Regarding Claim 3, Li in view of Dosovitskiy teaches the computing system of claim 1.
Li in view of Dosovitskiy does not teach distribute the plurality of multi-channel time- synchronized signals across a set of red, green and blue channels.
Le teaches distribute the plurality of multi-channel time- synchronized signals across a set of red, green and blue channels (Fig. 1, B. Spectrogram Module: 2D RCNNs, Le teaches taking two-lead input ECG signals and producing a spectrogram. The Examiner notes in Fig. 1 how the spectrogram is an RGB image, and therefore is analogous to the claimed “across a set of red, green and blue channels”.).
Li, Dosovitskiy, and Le are considered to be analogous to the claimed invention as they are in the same field of using neural network models to analyze image data. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Li in view of Dosovitskiy such that the input ECG signals taught by Li in view of Dosovitskiy are transformed into an RGB spectrogram as taught by Le. The motivation for this combination being the ability to retain more image from the input signal through the usage of three channels as opposed to a binary image only containing information in two channels.
Claims 9, 15, and 22 are the computer-readable storage medium, semiconductor apparatus, and method claims, respectively, corresponding to claim 3, and are similarly rejected.
Claims 4, 10, 16, and 23 are rejected as being unpatentable over Li in view of Dosovitskiy in view of Kobayashi (US 2009/0046947; hereinafter “Kobayashi”).
Regarding Claim 4, Li in view of Dosovitskiy teaches the computing system of claim 1.
Li in view of Dosovitskiy does not teach to normalize the plurality of image patches before the plurality of image patches are combined into the image.
Kobayashi teaches to normalize the plurality of image patches before the plurality of image patches are combined into the image (Fig. 6B, [0072-0080], Kobayashi teaches normalizing image data prior to combining them.).
Li, Dosovitskiy, and Kobayashi are considered to be analogous to the claimed invention as they are in the same field of image generation. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Li in view of Dosovitskiy such that the image patches, as taught by Li in view of Dosovitskiy, are normalized based prior to combining, based on the logic taught by Kobayashi. The motivation for this combination being the ability to standardize the images prior to combining them.
Claims 10, 16, and 23 are the computer-readable storage medium, semiconductor apparatus, and method claims, respectively, corresponding to claim 4, and are similarly rejected.
Claims 6, 12, 18, and 25 are rejected as being unpatentable over Li in view of Dosovitskiy in view of Arnab et al. (“ViViT: A Video Vision Transformer”, DOI: 10.1109/ICCV48922.2021.00676; hereinafter “Arnab”) in view of Ohmata et al. (“8K Time into Space: Web-Based Interactive Storyboard with Playback for Hundreds of Videos”, DOI: 10.1109/ISM.2016.0096; hereinafter “Ohmata”).
Regarding Claim 6, Li in view of Dosovitskiy teaches the computing system of claim 1.
Li in view of Dosovitskiy does not teach wherein the attention-based neural network is a video transformer neural network, and wherein the instructions, when executed, further cause the processor to: partition the image into a plurality of matrices; and aggregate the plurality of matrices into a video.
Arnab teaches wherein the attention-based neural network is a video transformer neural network (3. Video Vision Transformers, Fig. 1, Arnab teaches a ViT neural network which takes videos as input.),
Li, Dosovitskiy, and Arnab are considered to be analogous to the claimed invention as they are in the same field of processing images and video by neural network. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Li in view of Dosovitskiy such that ViT network taught by Li in view of Dosovitskiy is replaced by the video transformer neural network as taught by Arnab. The motivation for this combination being the ability to process video instead of only images, allowing for a larger amount of data to be processed.
Li in view of Dosovitskiy in view of Arnab does not teach wherein the instructions, when executed, further cause the processor to: partition the image into a plurality of matrices; and aggregate the plurality of matrices into a video.
Ohmata teaches wherein the instructions, when executed, further cause the processor to: partition the image into a plurality of matrices; and aggregate the plurality of matrices into a video (1) Animation of Sprite Sheet, Ohmata teaches a process of generating a sprite sheet image, that contains multiple sequential images side by side (wherein each sequential image is analogous to the claimed “matrices”), which is then converted from an image into a video.).
Li, Dosovitskiy, Arnab, and Ohmata are considered to be analogous to the claimed invention as they are in the same field of image processing. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Li in view of Dosovitskiy in view of Arnab such that the input image is divided into subsections and converted into a video using the logic taught by Ohmata. The motivation for this combination being the ability to use the original input images, and convert them into a video which can then be processed by the video transformer neural network.
Claims 12, 18, and 25 are the computer-readable storage medium, semiconductor apparatus, and method claims, respectively, corresponding to claim 6, and are similarly rejected.
Claims 19 is rejected as being unpatentable over Li in view of Dosovitskiy in view of Newton et al. (US 2021/0259778; hereinafter “Newton”).
Regarding Claim 19, Li in view of Dosovitskiy teaches the semiconductor apparatus of claim 13.
Li in view of Dosovitskiy is silent on wherein the logic coupled to the one or more substrates includes transistor channel regions that are positioned within the one or more substrates (The Examiner notes similarly to claim 1, that both Li and Dosovitskiy disclose image processing methods which are commonly known to be performed by computer machines, which include semiconductors.).
Newton teaches wherein the logic coupled to the one or more substrates includes transistor channel regions that are positioned within the one or more substrates ([0097], Newton teaches a signal processing unit which can process ECG signals, which is constructed on a semiconductor substrate which includes transistors configured as logic gates.).
Li, Dosovitskiy, and Newton are considered to be analogous to the claimed invention as they are in the same field of image and signal processing. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Li in view of Dosovitskiy such that the semiconductor apparatus is specifically configured in the way described by Newton. The motivation for this combination being the ability to configure of design of the semiconductor apparatus.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Natarajan et al., “Convolution-Free Wave Transformers for Multi-Lead ECG Classification, DOI: https://doi.org/10.48550/arXiv.2109.15129
Arjun et al., “Introducing Attention Mechanism for EEG Signals: Emotion Recognition with Vision Transformers”, DOI: 10.1109/EMBC46164.2021.9629837
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PROMOTTO TAJRIAN ISLAM whose telephone number is (703)756-5584. The examiner can normally be reached Monday - Friday 8:30 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PROMOTTO TAJRIAN ISLAM/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669