Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Amendment
This is in response to applicant’s amendment/response filed on 06/26/2026, which has been entered and made of record. Claims 1-8, 13-15, 19, 20 have been amended. No claim has been cancelled. No claim has been added. Claims 1-20 are pending in the application.
Response to Arguments
Applicant’s arguments on 06/26/206 have been fully considered but are moot because the arguments do not apply to any of the references being used in the current rejection.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 5-8, 10-15, 17, 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kothari et al. (US Pub 2021/0375023 A1) in view of Sun et al. (US Patent 12,485,898 B1).
As to claim 1, Kothari discloses one or more processors, comprising: circuitry to:
use a first portion of one or more neural networks to generate one or more portions of one or more images (Kothari, ¶0057, “a transformer can attempt to understand an entire story in order to generate appropriate animation.” “transformer 212 can have an ability to go back and modify already generated animation as transformer 212 better understands this story and importance of objects and events in that story. In at least one embodiment, transformer 212 can guide VAE 214 in generating objects required for this animation.”);
identify one or more objects within the generated one or more portions and within corresponding ground truth images (Kothari, ¶0048, “an image analysis component 114 can analyze this image data, such as to determine text included in this image data, as well as to determine changes to text represented in this data. In at least one embodiment, a content manager 116 can then utilize this text to generate AR content to be presented via headset 104. In at least one embodiment, this can include identifying objects, characters, or other elements represented in this text and generating images or animation representative of those objects, characters, or elements.” ¶0051, “one or more neural networks can be used attempt to animate an entire story or sequence represented in input text.” ¶0061, “use of cached latent space with this VAE enables extracted features of a scene, as captured by an AR camera, can be leveraged to build this cache, and groom a scene around that information. In at least one embodiment, if a real-world background includes a wall with a painting and text content also includes a painting, this real-world scene and object positioning can be leveraged while creating a clip.” ¶0348, “once imaging data 3108 is received, AI-assisted annotation 3110 may be used to aid in generating annotations corresponding to imaging data 3108 to be used as ground truth data for a machine learning model.” “labeled clinic data 3112 (e.g., annotations provided by a clinician, doctor, scientist, technician, etc.) may be used as ground truth data for training a machine learning model. In at least one embodiment, AI-assisted annotations 3110, labeled clinic data 3112, or a combination thereof may be used as ground truth data for training a machine learning model.”);
generate a task-based supervisory signal based, at least in part, on using a second portion of the one or more neural networks to compare the identified one or more objects within the generated one or more portions and within the corresponding ground truth images (Kothari, ¶0384, “model training 3114 may include retraining or updating an initial model 3304 (e.g., a pre-trained model) using new training data (e.g., new input data, such as customer dataset 3306, and/or new ground truth data associated with input data).” “parameters may be updated and re-tuned for a new data set based on loss calculations associated with accuracy of output or loss layer(s) at generating predictions on new, customer dataset” ¶0390, “customer dataset 3306 may be applied to initial model 3304 any number of times, and ground truth data may be used to update parameters of initial model 3304 until an acceptable level of accuracy is attained for refined model”); and
adjust weights of the first portion of the one or more neural networks (Kothari, ¶0097, “inference and/or training logic 615 may be used in system FIG. 8 for inferencing or predicting operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and/or architectures, or neural network use cases”)
Kothari does not explicitly disclose adjust weights of the first portion of the one or more neural networks based, at least in part, on an output of the second portion of the one or more neural networks from the comparison and the task-based supervisory signal.
However, comparing interim result to ground truth and adjusting weights of each node in neural network are well known in the art of artificial intelligence.
Sun teaches generate a task-based supervisory signal based, at least in part, on using a second portion of the one or more neural networks to compare the identified one or more objects within the generated one or more portions and within the corresponding ground truth images (Sun, Col 10, lines 57-65, “The road primitive module 214 may perform object detection to identify road primitives (e.g., cones, signs, lane markers, and other objects).” Col 11, lines 20-21, “the feature extractor may make use of a graphical neural network (GNN).” Col 12, lines 26, “the feature extractor may optimize multi-task training objective of the hierarchical graph network” Col 14, lines 61, “neural network-based object identification” Col 21, lines 26-29, “Each task may be performed by a separate machine learning model. Alternatively, a single machine learning model may perform each of the tasks or a subset of the tasks. Additionally, or alternatively, different machine learning models may be trained to perform different combinations of the tasks.” Col 24, lines 24-52, “the output may include one or more predictions or inferences. For example, an output prediction or inference may include scene encoded data, set of candidate target locations, candidate driving paths, and/or candidate driving path metrics. Processing logic may then compare the predicted or inferred output to one or more ground truth measurements (e.g., measured lane dimensions and trajectories) that may be included in the training data item. Processing logic determines an error (i.e., a classification error) based on the differences between the output of the machine learning model and the known classification (e.g., a target driving path compared to ground truth measurements).”); and
adjust weights of the first portion of the one or more neural networks based, at least in part, on an output of the second portion of the one or more neural networks from the comparison and the task-based supervisory signal (Col 24, lines 24-52, “Processing logic adjusts weights of one or more nodes in the machine learning model based on the error. An error term or delta may be determined for each node in the artificial neural network. Based on this error, the artificial neural network adjusts one or more of its parameters for one or more of its nodes (the weights for one or more inputs of a node). Parameters may be updated in a back propagation manner, such that nodes at a highest layer are updated first, followed by nodes at a next layer, and so on. An artificial neural network contains multiple layers of “neurons”, where each layer receives, as input, values from neurons at a previous layer. The parameters for each neuron include weights associated with the values that are received from each of the neurons at a previous layer. Accordingly, adjusting the parameters may include adjusting the weights assigned to each of the inputs for one or more neurons at one or more layers in the artificial neural network.”).
Kothari and Sun are considered to be analogous art because all pertain to deep learning. It would have been obvious before the effective filing date of the claimed invention to have modified Kothari with the features of “generate a task-based supervisory signal based, at least in part, on using a second portion of the one or more neural networks to compare the identified one or more objects within the generated one or more portions and within the corresponding ground truth images; and adjust weights of the first portion of the one or more neural networks based, at least in part, on an output of the second portion of the one or more neural networks from the comparison and the task-based supervisory signal.” as taught by Sun. All the claimed elements were known in the prior art and one skilled in the art could have combined the elements as claimed by known methods with no change in their respective functions, and the combination would have yielded predictable results to one of ordinary skill in the art at the time of the invention.
As to claim 2, claim 1 is incorporated and the combination of Kothari and Sun discloses the one or more neural networks generate the one or more portions of one or more images based, at least in part, on one or more loss values calculated by the one or more comparison (Kothari, ¶0384, “model training 3114 may include retraining or updating an initial model 3304 (e.g., a pre-trained model) using new training data (e.g., new input data, such as customer dataset 3306, and/or new ground truth data associated with input data). In at least one embodiment, to retrain, or update, initial model 3304, output or loss layer(s) of initial model 3304 may be reset, or deleted, and/or replaced with an updated or new output or loss layer(s). In at least one embodiment, initial model 3304 may have previously fine-tuned parameters (e.g., weights and/or biases) that remain from prior training, so training or retraining 3114 may not take as long or require as much processing as training a model from scratch. In at least one embodiment, during model training 3114, by having reset or replaced output or loss layer(s) of initial model 3304, parameters may be updated and re-tuned for a new data set based on loss calculations associated with accuracy of output or loss layer(s) at generating predictions on new, customer dataset 3306 (e.g., image data 3108 of FIG. 31).” Retraining can be part of neural network feature.).
As to claim 3, claim 1 is incorporated and combination of Kothari and Sun discloses the one or more neural networks include one or more transformer models each specific to one or more independent tasks (Kothari, ¶0057, “a segment of text can be encoded using layers of a convolutional neural network (CNN), as a CNN can determine an underlying pattern in this text, where different patterns can be used to generate different objects, actions, or occurrences in animation. In at least one embodiment, encoding from one or more CNNs 210 can be fed to a transformer 212, which can determine and understand how objects are related, which is important to bring consistency and coherence to a sequence of animation. In at least one embodiment, a transformer can attempt to understand an entire story in order to generate appropriate animation. In at least one embodiment, this can include determining which objects to emphasize, as well as a relative importance of various events to this overall story. In at least one embodiment, transformer 212 can have an ability to go back and modify already generated animation as transformer 212 better understands this story and importance of objects and events in that story.”).
As to claim 5, claim 1 is incorporated and combination of Kothari and Sun discloses the one or more neural networks are trained to perform the identification of the one or more object to include identifying text within the generated one or more portions and within corresponding ground truth images (Kothari, ¶0046, “an object within a field of view of a camera 106 of an AR headset 104 may include textual content. In at least one embodiment, this may include a book, magazine, digital interface, publication, or other source of text. In at least one embodiment, camera 106 can capture image or video data including a view of at least a portion of this text, and this text can be analyzed to determine content to be presented to user 102 through AR headset 104.” “an application can attempt to identify one or more objects represented or described in this text, and then generate image, video, or animation content to be displayed to user 102 via display 108. In at least one embodiment, as user 102 causes this text to change, such as my flipping pages of a book 150 or scrolling an interface, this generated content can update to represent this change in text.” ¶0053, “these images or video frames may be passed to an optical character recognition (OCR) application 206 or module that can generate text data corresponding to text represented in these images or video, wherein this text can be provided as an additional input to one or more neural networks to help improve inferencing results.”).
As to claim 6, claim 1 is incorporated and combination of Kothari and Sun discloses to perform modification of an image (Kothari, ¶0053, “these images or video frames may be passed to an optical character recognition (OCR) application 206 or module that can generate text data corresponding to text represented in these images or video, wherein this text can be provided as an additional input to one or more neural networks to help improve inferencing results.” ¶0054, “these text frames and any other identified text data can be used to generate image, animation, or other media content to be displayed or presented through AR device 202 or client device 204” ¶0057, “a segment of text can be encoded using layers of a convolutional neural network (CNN), as a CNN can determine an underlying pattern in this text, where different patterns can be used to generate different objects, actions, or occurrences in animation. In at least one embodiment, encoding from one or more CNNs 210 can be fed to a transformer 212, which can determine and understand how objects are related, which is important to bring consistency and coherence to a sequence of animation. In at least one embodiment, a transformer can attempt to understand an entire story in order to generate appropriate animation. In at least one embodiment, this can include determining which objects to emphasize, as well as a relative importance of various events to this overall story. In at least one embodiment, transformer 212 can have an ability to go back and modify already generated animation as transformer 212 better understands this story and importance of objects and events in that story.”).
As to claim 7, claim 1 is incorporated and combination of Kothari and Sun discloses to cause the one or more neural networks to be trained based, at least in part, on performing one or more independent tasks (Kothari, ¶0058, “a copy of an entire text can be obtained such that a system can have access to portions of that text other than are available from an AR device at a current time. In at least one embodiment, input text is pre-processed in order to detect sections that can be represented visually. In at least one embodiment, this can include using named entity recognition (NER) to detect objects and associated descriptions, as well as bi-directional encoder representations from transformers (BERT) to maintain context of these sections and group by relevance. In at least one embodiment, NER can be used to extract a real world noun entity from text data and classify this entity into a pre-defined category or classification, as may relate to a person, place, or object. In at least one embodiment, BERT can be used to perform transfer learning, sentiment analysis, sentence classification, and other such tasks useful for determining context. In at least one embodiment, NER can be performed unsupervised without labeled text using a BERT model that has been trained in an unsupervised fashion on a corpus with a specific language model objective.” ¶0067, ¶0084, ¶0384, “model training 3114 may include retraining or updating an initial model 3304 (e.g., a pre-trained model) using new training data (e.g., new input data, such as customer dataset 3306, and/or new ground truth data associated with input data). In at least one embodiment, to retrain, or update, initial model 3304, output or loss layer(s) of initial model 3304 may be reset, or deleted, and/or replaced with an updated or new output or loss layer(s).” ¶0384-0387.).
As to claim 8, combination of Kothari and Sun discloses a system, comprising: one or more processors to use a first portion of one or more neural networks to generate one or more portions of one or more images; identify one or more objects within the generated one or more portions and within corresponding ground truth images; generate a task-based supervisory signal based, at least in part, on using a second portion of the one or more neural networks to compare the identified one or more objects within the generated one or more portions and within the corresponding ground truth images; and adjust weights of the first portion of the one or more neural networks based, at least in part, on an output of the second portion of the one or more neural networks from the comparison and the task-based supervisory signal (See claim 1 for detailed analysis.).
As to claim 10, claim 8 is incorporated and combination of Kothari and Sun discloses the one or more neural networks include one or more transformer models each specific to one or more tasks (See claim 3 for detailed analysis.).
As to claim 11, claim 8 is incorporated and combination of Kothari and Sun discloses the one or more neural networks include a visual decoder, masking autoencoder, and visual encoder (See claim 4 for detailed analysis.).
As to claim 12, claim 8 is incorporated and combination of Kothari and Sun discloses the one or more neural networks include at least one neural network to perform one or more tasks of identifying text in the one or more portions of the one or more images (See claim 5 for detailed analysis.).
As to claim 13, claim 8 is incorporated and combination of Kothari and Sun discloses to perform modification of the one or more portions of the one or more images (See claim 6 for detailed analysis.).
As to claim 14, claim 8 is incorporated and combination of Kothari and Sun discloses the one or more processors are to train the one or more neural networks based, at least in part, on one or more results generated by the identification (See claim 7 for detailed analysis.).
As to claim 15, combination of Kothari and Sun discloses a method, comprising: using a first portion of one or more neural networks to generate one or more portions of one or more images; identifying one or more objects within the generated one or more portions and within corresponding ground truth images; generating a task-based supervisory signal based, at least in part, on using a second portion of the one or more neural networks to compare the identified one or more objects within the generated one or more portions and within corresponding ground truth images; and adjusting weights of the first portion of the one or more neural networks based, at least in part, on an output of the second portion of the one or more neural networks from the comparison and the task-based supervisory signal (See claim 1 for detailed analysis.).
As to claim 17, claim 15 is incorporated and combination of Kothari and Sun discloses the one or more neural networks include one or more transformer models each specific to one or more tasks (See claim 3 for detailed analysis.).
As to claim 19, claim 15 is incorporated and combination of Kothari and Sun discloses the one or more neural networks are each trained to perform one or more tasks including identifying text in an image (See claim 5 for detailed analysis.).
As to claim 20, claim 15 is incorporated and combination of Kothari and Sun discloses performing modification of an image (See claim 6 for detailed analysis.).
Claims 4, 9, 16, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Kothari et al. (US Pub 2021/0375023 A1) in view of Sun et al. (US Patent 12,485,898 B1) and He, Kaiming, et al. ("Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.).
As to claim 4, claim 1 is incorporated and Kothari discloses the one or more neural networks include a visual decoder, (Kothari, ¶0054, “this information can then be provided as input to one or more neural networks, such as one or more variational auto-encoders (VAEs). In at least one embodiment, these VAEs 214, which can include appropriate encoders 216 and decoders 218, can then generate images or animation that is meaningful with respect to these input text images, while also being consistent over a sequence or period of time.” ¶0058, “a copy of an entire text can be obtained such that a system can have access to portions of that text other than are available from an AR device at a current time. In at least one embodiment, input text is pre-processed in order to detect sections that can be represented visually. In at least one embodiment, this can include using named entity recognition (NER) to detect objects and associated descriptions, as well as bi-directional encoder representations from transformers (BERT) to maintain context of these sections and group by relevance.” ¶0059, “an autoencoder such as a VAE 214 can be utilized whose latent space is guided by an accumulating transformer 212, along with data in a cache”).
Kothari does not explicitly disclose masking autoencoder. However, Kothari discloses BERT (Kothari, ¶0058, “input text is pre-processed in order to detect sections that can be represented visually. In at least one embodiment, this can include using named entity recognition (NER) to detect objects and associated descriptions, as well as bi-directional encoder representations from transformers (BERT) to maintain context of these sections and group by relevance. In at least one embodiment, NER can be used to extract a real world noun entity from text data and classify this entity into a pre-defined category or classification, as may relate to a person, place, or object. In at least one embodiment, BERT can be used to perform transfer learning, sentiment analysis, sentence classification, and other such tasks useful for determining context. In at least one embodiment, NER can be performed unsupervised without labeled text using a BERT model that has been trained in an unsupervised fashion on a corpus with a specific language model objective. “). BERT is a well-known masked autoencoder.
He also teaches masking autoencoder (He, abstract, “masked autoencoders (MAE) are scalable self-supervised learners for computer vision.” Page 1, “The idea of masked autoencoders, a form of more general denoising autoencoders [58], is natural and applicable
in computer vision as well.“ Page 3, “Masked language modeling and its autoregressive counterparts, e.g., BERT [14] and GPT [47, 48, 4]”)
Kothari, Sun and He are considered to be analogous art because all pertain to deep learning. It would have been obvious before the effective filing date of the claimed invention to have modified Kothari with the features of “masking autoencoder” as taught by He. The suggestion/motivation would have been because the idea of masked autoencoders, a form of more general denoising autoencoders, is natural and applicable in computer vision as well (He, Page 1).
As to claim 9, claim 8 is incorporated and the combination of Kothari, Sun and He discloses the one or more neural networks generate the one or more portions of one or more images from an image masked based, at least in part, on a masked autoencoder (See claim 4 for detailed analysis.).
As to claim 16, claim 15 is incorporated and the combination of Kothari, Sun and He discloses discloses generating the one or more portions of the one or more images from one or more portions of one or more images masked based, at least in part, on a masked autoencoder (See claim 4 for detailed analysis. He, Fig. 1, Fig. 2. Page 3. Page 12. Fig. 11, Page 14.).
As to claim 18, claim 15 is incorporated and the combination of Kothari, Sun and He discloses discloses the one or more neural networks include a visual decoder, masking autoencoder, and visual encoder (See claim 4 for detailed analysis.).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Couleaud et al. (US Pub 2025/0078347 A1) discloses respective outputs 524 and 534 may be compared to ground truth values (e.g., annotated indications of the correct or desired outputs for given input(s)). In some embodiments, the image processing system may then adjust weights or other parameters of machine learning model 500 and 510 based on how closely the output corresponds to the ground truth value.
Jeong et al. (US Pub 2026/0105765 A1) discloses compare the predicted data with the ground truth data to determine a difference and train the AI model(s) 110c by adjusting the AI model's 110c internal weights and parameters proportional to the determined difference according to a loss function.
Xia et al. (US Pub 2026/0141729 A1) discloses Processing logic adjusts weights of one or more nodes in the machine learning model based on the error. An error term or delta may be determined for each node in the artificial neural network. Based on this error, the artificial neural network adjusts one or more of its parameters for one or more of its nodes (the weights for one or more inputs of a node), and adjusting the parameters may include adjusting the weights assigned to each of the inputs for one or more neurons at one or more layers in the artificial neural network.
Applicant's amendment necessitated the new ground(s) of rejection presented in this office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YU CHEN whose telephone number is (571)270-7951. The examiner can normally be reached on M-F 8-5 PST Mid-day flex.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached on 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YU CHEN/
Primary Examiner, Art Unit 2613
Prosecution Insights