Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The following is in response to the amendment filed on November 24, 2025.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-5, 8, 10-12, 15, 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Mishra et al (US PG Pub 2021/0397824), Provost et al (US PG Pub 2020/0075040) and Picard “Affective Computing”.
With respect to claim 1:
Mishra teaches:
A method, comprising:
Obtaining video sensor data from at least one senor associated with at least one user; (Paragraph [063], discloses retrieving videos)
Applying at least some of the video sensor data to a second machine learning model that analyzes a video sentiment of the at least one user to provide at least one video sentiment score indicating a sentiment of the at least one user based at least in part on the at least some of the video data; (Paragraph [082], discloses using a trained model to predict a sentiment score from the received video)
Applying the at least one audio sentiment score and the at least one video sentiment score to an ensemble machine learning model that determines an aggregate sentiment score of the at least one user based at least in part on the at least one audio sentiment score and the at least one video sentiment score; and (Paragraphs [116]-[119], disclose using a plurality of scores and aggregating them in an RNN to determine a facial prediction (aggregate score))
Initiating one or more automated remedial actions based at least in part on the aggregate sentiment score; wherein the method is performed by at least one processing device comprising a processor coupled to a memory. (Paragraph [060], discloses using the sentiment score to determine actions such as intention to commit fraud and therefore allow decision making based on that)
Mishra does not appear to explicitly disclose:
Obtaining audio sensor data;
Applying at least some of the audio sensor data to a first machine learning model that analyzes an audio sentiment of the at least one user to provide at least one audio sentiment score indicating a sentiment of the at least on user based at least in part on the at least some of the audio sensor data;
Wherein the one or more automated remedial actions comprise adjusting at least one parameter of a physical location associated with the at least one user to improve the aggregate sentiment score of the at least one user;
Provost teaches:
Obtaining audio sensor data; (Paragraph [019], discloses receiving audio data)
Applying at least some of the audio sensor data to a first machine learning model that analyzes an audio sentiment of the at least one user to provide at least one audio sentiment score indicating a sentiment of the at least on user based at least in part on the at least some of the audio sensor data; (Paragraph [06], discloses using a trained Machine Model to generate an emotion value from the trained audio)
It would have been obvious to one of ordinary skill in the art before the effective filing data of the present application to implement a method that utilized the teachings of Mishra and the teachings of Provost, both in the same filed of invention. This would allow for more accurate emotion evaluation.
The combination of Mishra and Provost does not appear to explicitly disclose:
Wherein the one or more automated remedial actions comprise adjusting at least one parameter of a physical location associated with the at least one user to improve the aggregate sentiment score of the at least one user;
Picard teaches:
Wherein the one or more automated remedial actions comprise adjusting at least one parameter of a physical location associated with the at least one user to improve the aggregate sentiment score of the at least one user; (Section 4.4, discloses adjusting physical factors such as lighting based on affective states)
It would have been obvious to one of ordinary skill in the art before the effective filing data of the present application to implement a method that utilized the teachings of Mishra in combination with Provost and the teachings of Picard, both in the same filed of invention. This would allow for the creation of appropriate moods in buildings.
With respect to claim 3:
Mishra teaches:
Preprocessing at least some of one or more of the audio sensor data and the video sensor data to satisfy one or more data processing criteria of one or more of the first machine learning model and the second machine learning model. (Paragraph [089], discloses preprocessing the received data)
With respect to claim 4:
Mishra teaches:
The preprocessing comprises one or more of: (i) selecting a number of audio features to send to the first machine learning model and (ii) detecting one or more human faces in the video sensor data and cropping one or more image frames using the detected one or more human faces. (Paragraph [089], discloses pre-processing the video data to determine and extract expressions)
With respect to claim 5:
Picard teaches:
Wherein the one or more automated remedial actions comprise one or more of: generating a notification, adjusting a temperature of a workspace area associated with the at least on user, adjusting a lighting of the workspace area associated with the at least one user, adjusting one or more of a volume and a content of music presented in the workspace area associated with the at least one user, and adjusting one or more scents provided in the workspace area associated with the at least one user. (Section 4.4, discloses adjusting lighting with respect to one user)
Claims 8, 10-12, 15, 17 and 18 are rejected according to claims 1, 3-5.
Claims 2, 9 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Mishra et al (US PG Pub 2021/0397824), Provost et al (US PG Pub 2020/0075040), Picard “Affective Computing” and Huang et al (US PG Pub 2022/0004754).
With respect to claim 2:
The combination of Mishra, Provost and Picard do not appear to explicitly disclose:
Providing an output of the ensemble model to at least one feedback agent that updates one or more of the first machine learning model and the second machine learning model.
Huang teaches:
Providing an output of the ensemble model to at least one feedback agent that updates one or more of the first machine learning model and the second machine learning model. (Paragraph 0054, "In embodiments, the programming instructions can instruct the at least one processor 507 to extract a sequence of historical image frames from each of the plurality of historical video clips, extract a stream of historical audio signals from each of the plurality of historical video clips and train the document identification model with the sequence of historical image frames and the stream of historical audio signals." Paragraph 0055, "In embodiments, the document identification model comprises a convolutional neural subnetwork and the programming instructions can instruct the at least one processor 507 to train the convolutional neural subnetwork with the sequence of historical image frames from each of the plurality of historical video clips." Paragraph 0056, "In embodiments, the document identification model comprises a recurrent neural subnetwork and the programming instructions can instruct the at least one processor 507 to train the recurrent neural subnetwork with the stream of historical audio signals from each of the plurality of historical video clips." Huang teaches that from the result of the identification score, the application will instruct one processor to extract historical image frames and historical audio signals to train the document identification model, which contains the two machine learning models. The processor is the feedback agent, and the document identification model is the model being updated. The model contains a recurrent neural subnetwork the train the recurrent neural subnetwork, thus training the models.)
Mishra, Provost, Picard and Huang are analogous to the claimed invention because they are all in the same field of invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Mishra, Provost and Picard to incorporate the teachings of Huang. This is because Huang teaches a more accurate way to determine the audio and video scores. Huang combines the audio and video scores into an ensemble model to output an aggregate sentiment score. Then, Huang uses that aggregate score to determine a remedial action.
Claims 9 and 16 are rejected according to claim 2.
Claims 6, 13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Mishra et al (US PG Pub 2021/0397824), Provost et al (US PG Pub 2020/0075040), Picard “Affective Computing” and Sundaresan et al (US PG Pub 2019/0087712).
With respect to claim 6:
The combination of Mishra, Provost and Picard do not appear to explicitly disclose:
One or more of the at least one audio sentiment score and the at least one video sentiment score comprises a score matrix indicating a probability score for each of a plurality of sentiment categories.
Sundaresan teaches:
One or more of the at least one audio sentiment score and the at least one video sentiment score comprises a score matrix indicating a probability score for each of a plurality of sentiment categories. (Paragraph 0027, "According to the present disclosure, multiple neural networks can analyze related data feeds to produce a score (i.e., an output). For example, a video feed can include images and audio. The video feed can be split into an image feed and an audio feed. One neural network can analyze the image feed to produce an image score. Another neural network can analyze the audio feed to produce an audio score." Paragraph 0047, "Scores 113, 123 can include one or more classification matrices. When the present disclosure refers to matrices, such matrices can be vectors (e.g., a matrix with a single column and/or a single row). A classification matrix can be an index of confidences, such as values representing probabilities. Each entry in the matrix can represent the NN's confidence in a certain outcome." Paragraph 0049, "First score 113 can be a matrix (e.g., a multi-dimensional vector) listing the confidence of each object in the first object set. Second score 123 can be a matrix listing the confidence of each object in the second object set." Sundaresan teaches that each neural network can produce an audio score and video score. These audio scores and video scores, which represent the probability of how confident they are, are then turned into matrices that also list their confidence levels).
Mishra, Provost, Picard and Sundaresan are analogous to the claimed invention because they are all in the same field of invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sundaresan to incorporate the teachings of the combination of Mishra, Provost, Picard because this leads to a more accurate result when determining the remedial action.
Claims 13 and 19 are rejected according to claim 6.
Claims 7, 14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Mishra et al (US PG Pub 2021/0397824), Provost et al (US PG Pub 2020/0075040), Picard “Affective Computing” and Gupta et al (US PG Pub 2021/0295423).
With respect to claim 7:
Mishra teaches: processing at least some of the video sensor data. (Paragraph [089])
The combination of Mishra, Provost and Picard do not appear to explicitly disclose:
To identify one or more user classes that are excluded from one or more of group meetings and group activities by evaluating pixel coordinates of at least some objects in a given image associated with users to identify the one or more excluded user classes.
Gupta teaches:
To identify one or more user classes that are excluded from one or more of group meetings and group activities by evaluating pixel coordinates of at least some objects in a given image associated with users to identify the one or more excluded user classes. (Paragraph 0019, "The image surfacing system can identify the product in user-uploaded images, isolate the product, and extract shape descriptors or feature parameters from the product in the user-uploaded images. The image surfacing system can analyze and compare the shape descriptors or feature parameters to group user-uploaded images with similar seller images. Furthermore, the image surfacing system can identify missing product perspectives from the seller images based on received user-submitted images and display user-submitted images depicting the product using the missing views." Gupta teaches that his image surfing system can analyze group user-uploaded images and identify missing product perspectives from the seller images. This is comparable to identifying missing user class images from a group photo by identifying the missing images from user-submitted images).
Mishra, Provost, Picard and Gupta are analogous to the claimed invention because they are all in the same field of invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gupta to incorporate the teachings of Mishra, Provost, Picard. This is because this would allow a more accurate analysis of the video images.
Claims 14 and 20 are rejected according to claim 7
Response to Arguments
Claim Rejections - 35 USC § 101
The instant amendments have overcome the 35 USC 101 rejection.
Claim Rejections - 35 USC § 103
Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARIELA D REYES whose telephone number is (571)270-1006. The examiner can normally be reached Monday-Friday, 7:30 am -5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Wiley can be reached at 571-272-3923. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Mariela Reyes/Supervisory Patent Examiner, Art Unit 2142