DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-6 of U.S. Patent No. 9,747,573. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant set of claims are a broader and obvious variation of the patented claims. For example, claim 1 of the application and patent are directed toward identifying emotional status based on emotion detected from a user, including facial emotion, and generating a parameter based on the emotional status, except, for example, the instant claim does not claim identifying facial emotions of multiple individuals.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-26 of U.S. Patent No. 10,496,947. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant set of claims are a slightly broader and obvious variation of the patented claims. For example, claim 1 of the application and patent are directed toward identifying emotional status based on emotion detected from a user, including facial emotion, and generating a parameter based on the emotional status, except that the instant claim 1 does not claim the work quality parameter which is later claimed in claim 3.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 9852328. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant set of claims are an obvious variation of the patented claims. For example, both the instant set of claims and the patent are directed toward identifying emotional status based on emotion detected from a user, including facial emotion, but the instant set of claims does not claim the audio aspect.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-22 of U.S. Patent No. 9,576,190. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant set of claims are broader variation of the patented claims. For example, both the instant set of claims and the patent are directed toward identifying emotional status based on emotion detected from a user, including facial emotion, but the instant claim 1 does not claim the negative emotion.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 10,235,562. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant set of claims are obvious variation of the patented claims. For example, both the instant set of claims and the patent are directed toward identifying emotional status based on emotion detected from a user, including facial emotion, but the instant claim 1 does not claim the negative emotion. Claim 2 of the patent similarly claims the virtual face mesh.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 10,255,488. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant set of claims are an obvious variation of the patented claims. For example, both the instant set of claims and the patent are directed toward identifying emotional status based on emotion detected from a user, including facial emotion.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 10,949,655. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant set of claims are an obvious variation of the patented claims. For example, claims of the instant application and that of the patent are directed toward identifying emotional status based on emotion detected from a user, including negative emotion based on detected images.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 10,963,679. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant set of claims are an obvious variation of the patented claims. For example, claims of the instant application and that of the patent are directed toward identifying emotional status based on emotion detected from a user, by mapping and aligning detected face to a virtual face mesh.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of US Patent 11,652,956. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant set of claims are an obvious variation of the reference claims. Both sets of claims are directed toward identifying emotional status based on emotion detected from a user, including facial emotion.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-13 of U.S. Patent No. 11,922,356. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant set of claims are broader version of the patented claims with wording variations. For example, the instant set of claims does not include the voice aspect of the invention claimed in the patent.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-10, 13-14, 16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Lau et al. (US 2013/0015946) in view of Conway et al. (US 9,269,374).
Claim 1
Lau teaches a computer-implemented method for workforce analytics, the method comprising: detecting an individual in one or more frames of a video stream ([0078] FIG. 10 is a schematic block diagram 1000 illustrating another example image capture method as can be used in embodiments of the disclosed technology. In FIG. 10, multiple images 1030, 1032, 1034, 1036 of a user 1030 are captured by a camera 1012 of a mobile device 1010);
dynamically determining, over the one or more frames, a deformation of a virtual face mesh of the individual ([0078] In the illustrated embodiment, the user is naturally changing and adjusting his position, facial expression, focus, and so on. In other embodiments, however, the user can be directed into different positions or facial expressions (e.g., by prompts on the image capture screen 1002) in order to produce a series of images with expected facial orientations and expressions (e.g., a side view, a front view, a smiling expression, a frowning expression, a stern expression, or any other such facial orientation or expression). [0109], detecting the face of the user, tracking one or more feature points on the detected face over multiple consecutive frames, estimating an affine transform between frames);
locating feature reference points of the individual ([0110] At 1422, facial components (also referred to as face landmark points) are located and extracted.);
aligning the virtual face mesh to the individual in one or more of the frames based at least in part on the feature reference points ([0111] At 1424, face alignment (or facial component alignment) is performed using the extracted facial components.);
determining that the deformation refers to at least one facial emotion selected from a plurality of reference facial emotions ([0074], different positions or facial expressions (e.g., by prompts on the image capture screen 1002) in order to produce a series of images with expected facial orientations and expressions (e.g., a side view, a front view, a smiling expression, a frowning expression, a stern expression, or any other such facial orientation or expression); [0078], different positions or facial expressions (e.g., by prompts on the image capture screen 1002) in order to produce a series of images with expected facial orientations and expressions (e.g., a side view, a front view, a smiling expression, a frowning expression, a stern expression, or any other such facial orientation or expression)… Furthermore, in embodiments in which a specific facial expression or facial orientation is captured in the one or more additional image, the additional images provide unique data points that will be matched to different (independent) reference images than one or more of the other captured images.);
Still Lau may not explicitly detail determining an emotional status for the individual based on the at least one facial emotion.
Conway teaches determining an emotional status for the individual based on the at least one facial emotion (Col. 12, lines 8-11, The video analysis module can further output time-coded video behavioral data that reflects the emotion(s) associated with the facial expression at any particular time during the communication for each face visible in the communication. Col. 16, lines 16-19, These may include instructions to analyze video and audio components of a user communication…extract emotions from the facial expressions in the video component,).
It would have been obvious to one of ordinary skill in the art before the filing date of the invention to incorporate emotion determination as taught by Conway with the image analysis method of Lau, because doing so would have provided video analytics that can understand a scene, and can qualify an object, understand the context around the object, and track the object through the scene. Commonly, video analytics detects changes occurring over successive frames of video, qualifies these changes in each frame, correlates qualified changes over multiple frames, and interprets these correlated changes (Col. 9, lines 1-7 of Conway).
Claim 2
The combination teaches the method of claim 1, further comprising establishing a video conference between the individual and a customer (Col, 3, lines 4; Col. 14, lines 27 of Conway, video communication).
Claim 3
The combination teaches the method of claim 1, further comprising: generating quality metrics including at least one work quality parameter associated with the individual based on the at least one facial emotion ([0074] of Lau, for example, smiling expression, angry expression, etc.; Col. 13, lines 47-60 of Conway, In some embodiments, distress and engagement data are aggregated with the speaker content and facial expression data. Col. 8, line 11-23, The control system 142 may also collect agent-specific unstructured and structured data including without limitation agent personality type…and any other agent data relevant to contact center performance. Additionally, one of ordinary skill in the art would recognize that the types of data collected by the contact center control system 142 that are identified above are simply examples and additional and/or different interaction data, user data, agent data, video data, and telephony data may be collected and processed by the control system 142. Col. 10, line 66- Col. 11, line 3, As further described herein, user data, agent data, and/or behavioral assessment of interaction data (e.g., personality type, linguistic analysis data, distress analysis data, video analysis data, etc.) may be employed to create more accurate predictive models for use in the contact center 100.).
Claim 4
The combination teaches the method of claim 3, further comprising recording the quality metrics of the individual in an employee record, wherein each of the quality metrics is time-stamped (Col. 1, lines 55-60 of Conway, The plurality of instructions include instructions, that when executed, receive a video communication from a user, wherein the video communication comprises an audio component and a video component; instructions, that when executed, analyze the video component to provide time-coded video behavioral data; Col. 8, lines 24-29 of Conway, The control system 142 may store recorded and collected interaction data in a database 152, including user data and agent data. In certain embodiments, agent data, such as agent scores for dealing with users, are updated daily. The control system 142 may store recorded and collected interaction data in a database 152.).
Claim 6
The combination teaches the method of claim 3, wherein the at least one work quality parameter includes a tiredness characteristic of the individual ([0078] of Lau, both-eyes-closed expression).
Claim 7
The combination teaches the method of claim 3, wherein the at least one work quality parameter includes a negative emotion characteristic of the individual ([0078] of Lau, frowning expression).
Claim 8
The combination teaches the method of claim 3, wherein the at least one work quality parameter includes a positive emotion characteristic of the individual ([0074] of Lau, smiling expression).
Claim 9
The combination teaches the method of claim 3, wherein the at least one work quality parameter includes a smile characteristic of the individual ([0074] of Lau, smiling expression).
Claim 10
The combination teaches the method of claim 1, wherein the determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions includes: comparing the at least one deformation of the virtual face mesh to reference facial parameters of the plurality of reference facial emotions; and selecting the facial emotion based on a comparison of the at least one deformation of the virtual face mesh to the reference facial parameters of the plurality of reference facial emotions ([0093] of Lau, In particular embodiments, and as illustrated in FIG. 11, the facial detection at 1112, the facial component detection and localization at 1114, and the feature descriptor generation at 1116 are performed in the enrollment phase, before a user requests authentication. In such embodiments, the resulting feature descriptors are stored for later reference and comparison with the feature descriptors of captured images.).
Claim 13
The combination teaches the method of claim 1, wherein the feature reference points include facial landmarks ([0110] of Lau, At 1422, facial components (also referred to as face landmark points) are located and extracted. ).
Claim 14
The combination teaches the method of claim 1, wherein the detecting of the individual includes applying a Viola-Jones algorithm to images associated with the individual ([0090] of Lau, FIG. 14, which uses an implementation of the Viola-Jones facial detector, can be used.).
Claim 16
The combination teaches the method of claim 1, wherein the aligning of the virtual face mesh is based on shape units associated with a face shape of the individual ([0110] of Lau, For example, in certain embodiments, eyes, nose, and mouth points are located. In particular, in certain implementations, corners of the user's mouth, the four corners of each of the user's eyes, and both nostrils of the user are extracted. [0137], In certain embodiments of the disclosed technology, the edges of the graph are represented by the mean distances between the different components of the training image set of the subject i… components of the i.sup.th subject, which are centered at the right eye, left eye, nose tip, right corner of the lips, and the left corner of the lips, respectively.).
Claim 18
The combination teaches the method of claim 1, wherein the plurality of facial emotions include at least one of: a neutral facial emotion, a positive facial emotion, or a negative facial emotion; wherein the positive facial emotion includes at least one of happiness, gratitude, kindness, or enthusiasm individual ([0074] of Lau, smiling expression); and wherein the negative facial emotion includes at least one of anger, stress, depression, frustration, embarrassment, irritation, sadness, indifference, confusion, or annoyance ([0078] of Lau, frowning expression).
Claims 19-20
These claims recite substantially the same limitations as those provided in claim 1 above, and therefore they are rejected for the same reasons.
Claims 5 are rejected under 35 U.S.C. 103 as being unpatentable over Lau et al. (US 2013/0015946) in view of Conway et al. (US 9,269,374) and Dimitriadis et al. (US 2014/0163960), hereinafter “Dimitri”.
Claim 5
The combination teaches the method of claim 3, except further comprising aggregating the quality metrics associated with the individual over a predetermined period to produce a work performance characteristic of the individual.
Dimitri teaches in [0062] The second layer of the device 200, as mentioned above, may be responsible for tracking the time-evolution of the emotional state 304 and the confidence score 306 of each of the segments 302.sub.1,2, . . . , n of the audio signal 300. The second layer may also be for defining an overall emotional state of the user that is determined in accordance with the current emotional state 402 that is tracked throughout each segment 302 of the audio signal 300. The overall emotional state may be determined in accordance with any of the above methods discussed with respect to the feature of combining a plurality emotional states 304.sub.1,2, . . . , n, or determined in accordance with any other known method. The current emotional state 402 may be defined in in real-time as the audio signal 300 may be processed in real-time. See also [0066]-[0067]-“ a predetermined number of the segments 302.sub.1,2, . . . , n within a predetermined time frame have an emotional state 304 of "angry"…the processor 208 may nonetheless change the current emotional state 402 to the emotional state 304 of the segments 302.sub.1,2, . . . , n based on the analysis of those segments 302.sub.1,2, . . . , n in total.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to incorporate determination of the value of work performance characteristic as taught by Dimitri with the facial analysis process of Lau in view of Conway, because doing so would have provided a way to monitor the current emotional state 402 of another user or party, such as a customer, in real-time. In this regard, the processor 208 of the device 200 may provide a user-detectable notification in response to determining that the current emotional state 402 of the audio signal 300 changes to another emotional state 304. The device 200 may, for example, provide a user-detectable notification on a display 212 of the device 200, as shown in FIG. 2. ([0068] of Dimitri).
Claims 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over Lau et al. (US 2013/0015946) in view of Conway et al. (US 9,269,374) and Kaneda (US 2011/0032378).
Claim 11
Lau discloses [0107]: Further, a learning algorithm can be applied to the weak classifiers using one or more reference images so that only the most important weak classifiers are selected based on their hit rate and miss rate. The weak classifiers can be further assembled into strong classifiers and applied in a cascade architecture in order to increase the speed with which the detection can be performed. [0115], The learning-based encoder can be trained using images of the user's face stored on the device (e.g., the reference images) or can be pre-trained by a much larger set of test images before it is used in the mobile device (e.g., pre-trained before implementation and storage in a mobile device). However, the combination fails to explicitly detail wherein the comparing of the at least one deformation of the virtual face mesh to reference facial parameters comprises applying a convolution neural network.
Kaneda teaches comparing of the at least one deformation of the virtual face mesh to reference facial parameters comprises applying a convolution neural network ([0044] The face detection unit 1001 detects the face of a person from an input image. This face detection can be performed by using a predetermined algorithm. As the predetermined algorithm, for example, a convolutional neural network is known, which hierarchically detects features including low-order features such as edges and high-order features such as eyes and a mouth and finally detects the barycentric position of the face).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to incorporate face detection algorithm as taught by Kaneda with the facial analysis process of Lau in view of Conway, because doing so would have provided a way to accurately recognize facial expressions even if parts such as eyes and mouths have similar shapes ([0009] of Kaneda).
Claim 12
The combination teaches the method of claim 10, except wherein the comparing of the at least one deformation of the virtual face mesh to reference facial parameters comprises applying a state vector machine.
Kaneda teaches in [0102]: it suffices to use, for example, the technique disclosed in K. Mori, M. Matsugu, and T. Suzuki, "Face Recognition Using SVM Fed with Intermediate Output of CNN for Face Detection".
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to incorporate face detection algorithm as taught by Kaneda with the facial analysis process of Lau in view of Conway, because doing so would have provided a way to accurately recognize facial expressions even if parts such as eyes and mouths have similar shapes ([0009] of Kaneda).
Claims 15 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Lau et al. (US 2013/0015946) in view of Conway et al. (US 9,269,374) and Wang et al. (US 2014/0043329).
Claim 15
The combination teaches the method of claim 1, except wherein the locating of the feature reference points includes applying an Active Shape Model algorithm to images associated with the individual.
Wang teaches the video processing involving Active Shape Model algorithm to images associated with the individual ([0043] Two classical methods for facial landmark detection processing are the Active Shape Model (ASM) and the Active Appearance Model (AAM). The ASM and AAM use statistical models trained from labeled data to capture the variance of shape and texture. The ASM is disclosed in "Statistical Models of Appearance for Computer Vision," by T. F. Cootes and C. F. Taylor, Imaging Science and Biomedical Engineering, University of Manchester, Mar. 8, 2004.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to incorporate the use of ASM as taught by Wang with the facial analysis process of Lau in view of Conway, because doing so would have provided a way for primary facial features such as eyes, mouth, and nose may be individually characterized. ([0022] of Wang).
Claim 17
The combination teaches the method of claim 16, further comprising: estimating intensities of the shape units associated with the face shape; estimating intensities of action units associated with face mimics; and estimating rotations of the virtual face mesh around three orthogonal axes and its translations along the axes.
Wang teaches in [0048] In an embodiment, seven facial landmark points for eyes, mouth and nose may be used, and may be modeled by seven parameters: three rotation parameters, two translation parameters, one scale parameter, and one mouth width parameter.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to incorporate estimation of parameters as taught by Wang with the facial analysis process of Lau in view of Conway, because doing so would have provided a way for primary facial features such as eyes, mouth, and nose may be individually characterized. ([0022] of Wang).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THOMAS H MAUNG whose telephone number is (571)270-5690. The examiner can normally be reached Monday-Friday, 9am-6pm, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Carolyn R. Edwards can be reached at 1-(571) 2707136. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THOMAS H MAUNG/Primary Examiner, Art Unit 2692
/CAROLYN R EDWARDS/Supervisory Patent Examiner, Art Unit 2692
/CAROLYN R EDWARDS/Supervisory Patent Examiner, Art Unit 2692