DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 have been examined.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/05/2025 has been entered.
Response to Amendment
Claims 1, 15, and 19 have been amended.
Applicant’s arguments with respect to claims 1, 15, and 19 regarding the new limitations: “the semantic memory comprises a context of a previous indication of previously detected confidential content”, have been considered but are moot of the new ground of rejection presented in the current office action.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 15 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 15 recites the limitation "the first confidential content" in the underlined portion of the claim. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-3, 5-10, and 12-20 are rejected under 35 U.S.C. 103 as being unpatentable over prior art of record US 20210051294 to Roedel et al (hereinafter Roedel), Context-Aware Data Loss Prevention for Cloud Storage Services by Ong et al (hereinafter Ong) and prior art of record US 20230222236 to Devarao et al (hereinafter Devarao).
As per claim 1, Roedel teaches:
A system comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations comprising:
receiving an indication of a conferencing session having a plurality of participants (Roedel: [0057]: the communication session server 116 facilitates the establishment of a communication session between two or more of the client devices 104-114. The communication session may also be a video stream (with or without audio). [0080]: The client device 104 and the communication session server 116 may negotiate a communication protocol by which to transfer the content 218, and the communication session server 116 which remote devices are to receive a video, or plurality of images, of the content 218. For example, the user of the client device 104 may designate and/or invite or more user accounts to a communication session, and the communication session server 116 may then send an invitation to the invited user accounts to attend the communication session with the user of the client device 104);
determining a first user confidentiality level of a first participant of the plurality of participants (Roedel: [0091]. [0093]: To determine the security clearances of the user accounts viewing or participating in the communication session, the obfuscation module 216 may query the communication session server 116 to provide this information. The obfuscation module 216 may query the communication session server 116 at predetermined instances during the communication session. [0094] In response to the query from the obfuscation module 216, the communication session server 116 may reference a user accounts database 326 (illustrated in FIG. 3) to obtain the security clearance value associated with a particular user account (e.g., using a username or other user identifier to query the user accounts database 326) (first participant). The communication session server 116 then communicates the obtained security clearance values to the obfuscation module 216);
receiving a first conference input of the conferencing session, wherein the first conference input represents a first content type (Roedel: [0078] Where the communication session involves video (with or without audio), the video may be stored as content 218. The content 218 may also include a segment or portion of a live video stream being communicated by the communication client 212);
identifying, based on the first content type, first confidential content by evaluating the first conference input using a (Roedel: [0006]: Training the obfuscation model may include training the obfuscation model to recognize the words and/or images that appear in the uploaded electronic files. In this embodiment, the uploaded electronic files are considered confidential to the organization. [0009]: The obfuscation model may then be used to detect whether an object appearing in an image or video frame is likely to be a confidential object and, further still, which level of confidentiality likely applies to the detected object. [0013]: As the computing device is processing the live video stream, the computing device may employ an image recognition algorithm on the video frames of the live video stream to determine whether the video frames of the live video stream contain an object to be obfuscated. In applying the image recognition algorithm, the computing device may leverage the obfuscation model)
determining a first content confidentiality level associated with the detected first confidential content (Roedel: [0009]: The obfuscation model may then be used to detect whether an object appearing in an image or video frame is likely to be a confidential object and, further still, which level of confidentiality likely applies to the detected object. [0091]: The obfuscation module 216 may determine the security clearance required to view the detected object by referencing the obfuscation model 224 and obtaining a likely confidentiality attribute value associated with the detected object. Thus, through a machine-learning algorithm, the obfuscation model 224 also outputs a matrix or other data structure that lists the confidentiality attribute values potentially associated with the detected object, and the likelihood that the detected object should be classified with a particular confidentiality attribute value);
generating, based on the first user confidentiality level of the first participant and the first content confidentiality level of the first confidential content, a first cloaked conference output by automatically modifying the detected first confidential content in the first conference input according to the first content confidentiality level (Roedel: [0091]: the obfuscation module 216 may be configured to selectively obfuscate the objects appearing within the content 218 based on one or more user account(s) present in the communication session and in communication with the client device 104. The obfuscation module 216 may be configured to determine a security clearance or authorization clearance required to view the detected object, and then determine which of the user account(s) present in the communication session have the requisite security clearance to view the detected object. The obfuscation module 216 may determine the security clearance required to view the detected object by referencing the obfuscation model 224 and obtaining a likely confidentiality attribute value associated with the detected object. [0095] During the transmission of the communication session, the obfuscation module 216 then compares the received security clearance values with the likely security clearance determined via the obfuscation module 216. The obfuscation module 216 may then manipulate or modify the duplicated video frame(s) to create the obfuscated image(s) 220. Those user account(s) that are not authorized to view the detected object without obfuscation can view the content 218 with the detected object being obfuscated); and
broadcasting the first cloaked conference output to as a part of the conferencing session to second participant, wherein the second participant corresponds to a second user confidentiality level, and the second user confidentiality level is lower than the first content confidentiality level (Roedel: [0095]: the client device 104 transmits a content 218 for each client device in communication with the client device 104 and party to the communication session. Those user account(s) that are not authorized to view the detected object without obfuscation can view the content 218 with the detected object being obfuscated).
Roedel teaches a machine learning model to detect confidential content but does not teach: using a combination of a trained multimodal machine learning (ML) model and a semantic memory, the semantic memory comprises a context of a previous indication of previously detected confidential content, and and the first confidential content is based on the context as recalled from the semantic memory. However, Ong teaches:
using a combination of a trained machine learning (ML) model and a semantic memory, the semantic memory comprises a context of a previous indication of previously detected confidential content, and the first confidential content is based on the context as recalled from the semantic memory (Ong: page 403: right column: 2) Bidirectional LSTM Networks: To better utilize the context of a token, we leverage the temporal dependencies among tokens in both directions (forward and backward) through a bidirectional LSTM network [22]. We use both the past features (via forward states) and future features (via backward states) for a specific sequence window. The bidirectional LSTM network is illustrated in Figure 5. As we can see, the contextual information of a word can be leveraged through the learning in the hidden layer which consists of both forward and backward passes. Therefore, the sensitivity classification of a token does not only depend on the features of that token, but is also conditioned on other words present within the context. Page 404: right column: C. Sentence Level In our sentence level experiment, we have trained three different models: BOW, LSTM, and CNN. For this experiment, we manually extract sentences from the documents and label them. To make sure there is sufficient contextual information in each sentence, we only consider sentences which have at least five tokens. Page 405: left and right columns: The word embedding feature vectors are provided as input to the LSTM network with 64 output dimensions. We run 5 epochs for training both LSTM and CNN. D. Token Level: In the token level experiment, we have trained two models: the standard many-to-many LSTM and Bidirectional LSTM (Bi-LSTM). Our dataset contains a total of 9,427 tokens, with only 390 marked as sensitive. The size of word embedding in both LSTM and Bi-LSTM is set to be 128. The embedding feature vectors are then provided as input to the LSTM and Bi-LSTM networks with 64 output dimensions. We use the same setting as the LSTM in sentence level detection. By utilizing the contextual information of target tokens through both forward and backward passes, Bi-LSTM has shown superior overall performance among all the methods, i.e., the context of sensitive information is learned and stored during training. Page 406: left column: VII. CONCLUSION: In this paper, we have introduced several model based methods for detecting sensitive content in the hybrid cloud by leveraging the contextual semantic information through advanced deep learning models. The developed data loss prevention system can detect sensitive content at various degrees of granularity within the document, and immediately respond to any behavior or events violating data sensitivity compliance, i.e., sensitive content is identified based on the context that is recalled from the LSTM network).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to employ the teachings of Ong in the invention of Roedel to include the above limitations. The motivation to do so would be to provide real-time sensitivity detection at various hierarchical degrees of granularity (Ong: pag4 400: left column: paragraph 2).
Roedel in view of Ong does not teach: a trained multimodal machine learning (ML) model. However, Devarao teaches:
a trained multimodal machine learning (ML) model (Devarao: [0023]: CACP may operate by detecting the presence of a screen sharing session and may monitor for and identify specific content elements in a video stream. The monitoring of content elements may be based on performing artificial intelligence operations on the video stream to identify the specific content elements. The specific content elements may be imagery or text that should not be shared or is unintentionally shared by a sharer. For example, the content elements may include private (confidential) information such as usernames, passwords, account numbers, or other information. [0025]: CACP may leverage a machine learning (“ML”) model that operates as a multimodal model (“MMM”) that inputs not only images of screen usage, but also text input from a user, and application statuses of an active computer system. The multimodal model may be configured to identify content that should not be shared as part of the screen sharing session with more accuracy. [0026] CACP may operate by altering a video stream such that certain information is no longer viewable. [0077] In some embodiments, features of the objects may be determined using a supervised machine learning model built using training data).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to employ the teachings of Devarao in the invention of Roedel in view of Ong to include the above limitations. The motivation to do so would be because an MMM may operate to identify a particular content element with a higher confidence score as compared to a single modal ML model (Devarao: [0025]).
As per claim 2, Roedel in view of Ong and Devarao teaches:
The system of claim 1, further comprising: broadcasting the first conference input to the second participant having a higher or equal user confidentiality level than the first content confidentiality level, wherein the first conference input is broadcast unmodified (Roedel: [0095]: During the transmission of the communication session, the obfuscation module 216 then compares the received security clearance values with the likely security clearance determined via the obfuscation module 216. Where a security clearance value meets or exceeds the likely security clearance, the obfuscation module 216 records that the corresponding user account is authorized to view the object in the content 218. The client device 104 transmits a content 218 for each client device in communication with the client device 104 and party to the communication session. In this manner, those user account(s) that are authorized to view the detected object without obfuscation can do so).
As per claim 3, Roedel in view of Ong and Devarao teaches:
The system of claim 1, wherein the first conference input comprises at least a second content type, and wherein the multimodal ML model evaluates the first conference input based on the first content type and the second content type to detect the first confidential content (Devarao: [0025]: CACP may leverage a machine learning (“ML”) model that operates as a multimodal model (“MMM”) that inputs not only images of screen usage, but also text input from a user, and application statuses of an active computer system. The multimodal model may be configured to identify content that should not be shared as part of the screen sharing session with more accuracy).
The examiner provides the same rationale to combine prior arts Roedel in view of Ong and Devarao as in claim 1 above.
As per claim 5, Roedel in view of Ong and Devarao teaches:
The system of claim 1, wherein the first content type is a video content type, and wherein modifying the first conference input comprises obscuring image data associated with the detected first confidential content (Roedel: [0088] The obfuscation module 216 is further configured to obfuscate an object detected in the video frame using different methodologies. In this regard, the methodologies may include applying an image manipulation or image modification technique to a portion of the video frame, such as a Gaussian blur, blocking, pixelating, or any other image manipulation or image modification technique that results in the obfuscation of the video frame or portion of the video frame).
As per claim 6, Roedel in view of Ong and Devarao teaches:
The system of claim 5, wherein the obscuring image data further comprises infilling pixel data associated with the detected first confidential content to match proximal pixel data of the first conference input (Roedel: [0088]: In this regard, the methodologies may include applying an image manipulation or image modification technique to a portion of the video frame, such as a Gaussian blur, blocking, pixelating, or any other image manipulation or image modification technique that results in the obfuscation of the video frame or portion of the video frame. In another embodiment, the obfuscation module 216 replaces the colors of particular pixels (e.g., those pixels corresponding to the object to be obfuscated) with a predetermined color (e.g., white, black, red, etc.). The pixel color replacement may be performed at the bit level to ensure that the information conveyed by the object and shown in the video frame is obfuscated).
As per claim 7, Roedel in view of Ong and Devarao teaches:
The system of claim 5, wherein the obscuring image data further comprises blurring pixel data associated with the detected first confidential content (Roedel: [0023] In a further embodiment of the method, obfuscating the portion of the first content data comprises blurring the portion of the first content data. [0129]: As also discussed above, the obfuscation module 216 may apply various image processing and/or image manipulation techniques to obfuscate the detected object, such as edge detection (to detect the edges of the object), a Gaussian blur (to blur the detected object or contents displayed by the detected object), blocking (to block the detected object or contents displayed by the detected object), pixelation (to pixelate the detected object or contents displayed by the detected object), or any other such image processing and/or image manipulation technique that results in the obfuscation of the detected object).
As per claim 8, Roedel in view of Ong and Devarao teaches:
The system of claim 3, wherein the first content type and the second content type are different (Devarao: [0025]: CACP may leverage a machine learning (“ML”) model that operates as a multimodal model (“MMM”) that inputs not only images of screen usage, but also text input from a user, and application statuses of an active computer system. The multimodal model may be configured to identify content that should not be shared as part of the screen sharing session with more accuracy).
The examiner provides the same rationale to combine prior arts Roedel in view of Ong and Devarao as in claim 1 above.
As per claim 9, Roedel in view of Ong and Devarao teaches:
The system of claim 1, further comprising:
receiving a second conference input associated with the conferencing session, wherein the second conference input is received after the first conference input (Roedel: [0078]: Where the communication session involves video (with or without audio), the video may be stored as content 218. The content 218 may also include a segment or portion of a live video stream being communicated by the communication client 212) ;
evaluating the second conference input using the multimodal ML model to detect second confidential content (Roedel: [0079]: the obfuscation module 216 determines whether each video frame includes a feature indicating that an object is present in the video frame. The feature may include content that should be obfuscated and/or includes an obfuscation marker that indicates that a particular object or content should be obfuscated. Devarao: [0025]: CACP may leverage a machine learning (“ML”) model that operates as a multimodal model (“MMM”) that inputs not only images of screen usage, but also text input from a user, and application statuses of an active computer system. The multimodal model may be configured to identify content that should not be shared as part of the screen sharing session with more accuracy);
determining a second content confidentiality level associated with the detected second confidential content (Roedel: [0091]: The obfuscation module 216 may determine the security clearance required to view the detected object by referencing the obfuscation model 224 and obtaining a likely confidentiality attribute value associated with the detected object. Thus, through a machine-learning algorithm, the obfuscation model 224 also outputs a matrix or other data structure that lists the confidentiality attribute values potentially associated with the detected object, and the likelihood that the detected object should be classified with a particular confidentiality attribute value);
comparing a respective user confidentiality level of each participant to the second content confidentiality level (Roedel: [0095] During the transmission of the communication session, the obfuscation module 216 then compares the received security clearance values with the likely security clearance determined via the obfuscation module 216);
generating a second cloaked conference output by automatically modifying the detected second confidential content in the second conference input (Roedel: [0095]: The obfuscation module 216 may then manipulate or modify the duplicated video frame(s) to create the obfuscated image(s) 220. When the content 218 (or the portion of the content 218 that is buffered in the memory) is transmitted to one or more of the client devices 106-114, the client device 104 may transmit a video where the object appears obfuscated in one or more of the video frame(s) and a video where the object does not appear obfuscated in one or more of the video frame(s)); and
broadcasting the second cloaked conference output to the second participant having a lower user confidentiality level than the second content confidentiality level (Roedel: [0095]: the client device 104 transmits a content 218 for each client device in communication with the client device 104 and party to the communication session. In this manner, those user account(s) that are authorized to view the detected object without obfuscation can do so, and those user account(s) that are not authorized to view the detected object without obfuscation can view the content 218 with the detected object being obfuscated).
The examiner provides the same rationale to combine prior arts Roedel in view of Ong and Devarao as in claim 1 above.
As per claim 10, Roedel in view of Ong and Devarao teaches:
The system of claim 9, wherein the second conference input comprises a third content type (Devarao: [0025]: CACP may leverage a machine learning (“ML”) model that operates as a multimodal model (“MMM”) that inputs not only images of screen usage, but also text input from a user, and application statuses of an active computer system. The multimodal model may be configured to identify content that should not be shared as part of the screen sharing session with more accuracy).
The examiner provides the same rationale to combine prior arts Roedel in view of Ong and Devarao as in claim 1 above.
As per claim 12, Roedel in view of Ong and Devarao teaches:
The system of claim 1, wherein automatically modifying the detected first confidential content in the first conference input occurs in near real-time (Roedel: [0013]: Using a computing device, a user may initiate a communication session, such as a live video stream, with another computing device in communication with the communication session server. As the computing device is processing the live video stream, the computing device may employ an image recognition algorithm on the video frames of the live video stream to determine whether the video frames of the live video stream contain an object to be obfuscated. [0014] When an object, or a confidentiality marker associated with content marked for obfuscation, is detected according to the obfuscation model and/or the template matching, the computing device may then obfuscate the object).
As per claim 13, Roedel in view of Ong and Devarao teaches:
The system of claim 1, wherein automatically modifying the detected first confidential content is performed by the multimodal ML model (Devarao: [0025]: CACP may leverage a machine learning (“ML”) model that operates as a multimodal model (“MMM”) that inputs not only images of screen usage, but also text input from a user, and application statuses of an active computer system. The multimodal model may be configured to identify content that should not be shared as part of the screen sharing session with more accuracy. [0026] CACP may operate by altering a video stream such that certain information is no longer viewable. Altering of the video stream may include smearing, smudging, blurring, removing, deemphasizing, or obscuring the video stream in whole or in part).
The examiner provides the same rationale to combine prior arts Roedel in view of Ong and Devarao as in claim 1 above.
As per claim 14, Roedel in view of Ong and Devarao teaches:
The system of claim 1, wherein automatically modifying the detected first confidential content is performed by a different multimodal ML model (Devarao: [0025]: CACP may leverage a machine learning (“ML”) model that operates as a multimodal model (“MMM”) that inputs not only images of screen usage, but also text input from a user, and application statuses of an active computer system. The multimodal model may be configured to identify content that should not be shared as part of the screen sharing session with more accuracy. [0026] CACP may operate by altering a video stream such that certain information is no longer viewable. Altering of the video stream may include smearing, smudging, blurring, removing, deemphasizing, or obscuring the video stream in whole or in part).
The examiner provides the same rationale to combine prior arts Roedel in view of Ong and Devarao as in claim 1 above.
As per claim 15, Roedel teaches:
A method of preventing disclosure of confidential content in a conferencing session, comprising:
receiving an indication of the conferencing session having a plurality of participants (Roedel: [0057]: the communication session server 116 facilitates the establishment of a communication session between two or more of the client devices 104-114. The communication session may also be a video stream (with or without audio). [0080]: The client device 104 and the communication session server 116 may negotiate a communication protocol by which to transfer the content 218, and the communication session server 116 which remote devices are to receive a video, or plurality of images, of the content 218. For example, the user of the client device 104 may designate and/or invite or more user accounts to a communication session, and the communication session server 116 may then send an invitation to the invited user accounts to attend the communication session with the user of the client device 104);
determining a first user confidentiality level of a first participant and a second user confidentiality level of a second participant of the plurality of participants (Roedel: [0091]. [0093]: To determine the security clearances of the user accounts viewing or participating in the communication session, the obfuscation module 216 may query the communication session server 116 to provide this information. The obfuscation module 216 may query the communication session server 116 at predetermined instances during the communication session. [0094] In response to the query from the obfuscation module 216, the communication session server 116 may reference a user accounts database 326 (illustrated in FIG. 3) to obtain the security clearance value associated with a particular user account (e.g., using a username or other user identifier to query the user accounts database 326). The communication session server 116 then communicates the obtained security clearance values to the obfuscation module 216);
receiving a conference input of the conferencing session, wherein the conference input represents a first content type (Roedel: [0078] Where the communication session involves video (with or without audio), the video may be stored as content 218. The content 218 may also include a segment or portion of a live video stream being communicated by the communication client 212);
evaluating the conference input using a (Roedel: [0006]: Training the obfuscation model may include training the obfuscation model to recognize the words and/or images that appear in the uploaded electronic files. In this embodiment, the uploaded electronic files are considered confidential to the organization. [0009]: The obfuscation model may then be used to detect whether an object appearing in an image or video frame is likely to be a confidential object and, further still, which level of confidentiality likely applies to the detected object. [0013]: As the computing device is processing the live video stream, the computing device may employ an image recognition algorithm on the video frames of the live video stream to determine whether the video frames of the live video stream contain an object to be obfuscated. In applying the image recognition algorithm, the computing device may leverage the obfuscation model),
generating, based on the first user confidentiality level and a content confidentiality level of the one or more portions of the confidential content, a first modified conference output by automatically modifying a first portion of the detected confidential content (Roedel: [0079]: the obfuscation module 216 determines whether each video frame includes a feature indicating that an object is present in the video frame. [0095]: During the transmission of the communication session, the obfuscation module 216 then compares the received security clearance values with the likely security clearance determined via the obfuscation module 216. Where a security clearance value meets or exceeds the likely security clearance, the obfuscation module 216 records that the corresponding user account is authorized to view the object in the content 218. The obfuscation module 216 may then manipulate or modify the duplicated video frame(s) to create the obfuscated image(s) 220. When the content 218 (or the portion of the content 218 that is buffered in the memory) is transmitted to one or more of the client devices 106-114, the client device 104 may transmit a video where the object appears obfuscated in one or more of the video frame(s) and a video where the object does not appear obfuscated in one or more of the video frame(s), i.e., a first video frame comprising an object to be obfuscated is evaluated based on security clearance of the user accounts and the confidentiality attribute associated with the object and the object is obfuscated only for a first user account that does not have the requisite security clearance to view the object);
generating, based on the second user confidentiality level, a second modified conference output by automatically modifying a second portion of the detected confidential content (Roedel: [0079]: the obfuscation module 216 determines whether each video frame includes a feature indicating that an object is present in the video frame. [0095]: During the transmission of the communication session, the obfuscation module 216 then compares the received security clearance values with the likely security clearance determined via the obfuscation module 216. Where a security clearance value meets or exceeds the likely security clearance, the obfuscation module 216 records that the corresponding user account is authorized to view the object in the content 218. The obfuscation module 216 may then manipulate or modify the duplicated video frame(s) to create the obfuscated image(s) 220. When the content 218 (or the portion of the content 218 that is buffered in the memory) is transmitted to one or more of the client devices 106-114, the client device 104 may transmit a video where the object appears obfuscated in one or more of the video frame(s) and a video where the object does not appear obfuscated in one or more of the video frame(s), i.e., a second video frame comprising an object to be obfuscated is evaluated based on security clearance of a user account and the confidentiality attribute associated with the object and the object is obfuscated only for a second user account that does not have the requisite security clearance to view the object); and
broadcasting the first modified conference output to the first participant and the second modified conference output to the second participant (Roedel: [0095]: the client device 104 transmits a content 218 for each client device in communication with the client device 104 and party to the communication session. In this manner, those user account(s) that are authorized to view the detected object without obfuscation can do so, and those user account(s) that are not authorized to view the detected object without obfuscation can view the content 218 with the detected object being obfuscated).
Roedel teaches a machine learning model to detect confidential content but does not teach: evaluating using a combination of a trained multimodal machine learning (ML) model and a semantic memory to identify confidential content by detecting one or more portions of confidential content in the input based on a context as recalled from the semantic memory, wherein the semantic memory comprises a context of a previous indication of previously detected confidential content, and the first confidential content is based on the context as recalled from the semantic memory. However, Ong teaches:
evaluating using a combination of a trained machine learning (ML) model and a semantic memory to identify confidential content by detecting one or more portions of confidential content in the input based on a context as recalled from the semantic memory, wherein the semantic memory comprises a context of a previous indication of previously detected confidential content, and the first confidential content is based on the context as recalled from the semantic memory (Ong: page 403: right column: 2) Bidirectional LSTM Networks: To better utilize the context of a token, we leverage the temporal dependencies among tokens in both directions (forward and backward) through a bidirectional LSTM network [22]. We use both the past features (via forward states) and future features (via backward states) for a specific sequence window. The bidirectional LSTM network is illustrated in Figure 5. As we can see, the contextual information of a word can be leveraged through the learning in the hidden layer which consists of both forward and backward passes. Therefore, the sensitivity classification of a token does not only depend on the features of that token, but is also conditioned on other words present within the context. Page 404: right column: C. Sentence Level In our sentence level experiment, we have trained three different models: BOW, LSTM, and CNN. For this experiment, we manually extract sentences from the documents and label them. To make sure there is sufficient contextual information in each sentence, we only consider sentences which have at least five tokens. Page 405: left and right columns: The word embedding feature vectors are provided as input to the LSTM network with 64 output dimensions. We run 5 epochs for training both LSTM and CNN. D. Token Level: In the token level experiment, we have trained two models: the standard many-to-many LSTM and Bidirectional LSTM (Bi-LSTM). Our dataset contains a total of 9,427 tokens, with only 390 marked as sensitive. The size of word embedding in both LSTM and Bi-LSTM is set to be 128. The embedding feature vectors are then provided as input to the LSTM and Bi-LSTM networks with 64 output dimensions. We use the same setting as the LSTM in sentence level detection. By utilizing the contextual information of target tokens through both forward and backward passes, Bi-LSTM has shown superior overall performance among all the methods, i.e., the context of sensitive information is learned and stored during training. Page 406: left column: VII. CONCLUSION: In this paper, we have introduced several model based methods for detecting sensitive content in the hybrid cloud by leveraging the contextual semantic information through advanced deep learning models. The developed data loss prevention system can detect sensitive content at various degrees of granularity within the document, and immediately respond to any behavior or events violating data sensitivity compliance, i.e., sensitive content is identified based on the context that is recalled from the LSTM network).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to employ the teachings of Ong in the invention of Roedel to include the above limitations. The motivation to do so would be to provide real-time sensitivity detection at various hierarchical degrees of granularity (Ong: pag4 400: left column: paragraph 2).
Roedel in view of Ong does not teach: a trained multimodal machine learning (ML) model. However, Devarao teaches:
a trained multimodal machine learning (ML) model (Devarao: [0023]: CACP may operate by detecting the presence of a screen sharing session and may monitor for and identify specific content elements in a video stream. The monitoring of content elements may be based on performing artificial intelligence operations on the video stream to identify the specific content elements. The specific content elements may be imagery or text that should not be shared or is unintentionally shared by a sharer. For example, the content elements may include private (confidential) information such as usernames, passwords, account numbers, or other information. [0025]: CACP may leverage a machine learning (“ML”) model that operates as a multimodal model (“MMM”) that inputs not only images of screen usage, but also text input from a user, and application statuses of an active computer system. The multimodal model may be configured to identify content that should not be shared as part of the screen sharing session with more accuracy. [0026] CACP may operate by altering a video stream such that certain information is no longer viewable. [0077] In some embodiments, features of the objects may be determined using a supervised machine learning model built using training data).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to employ the teachings of Devarao in the invention of Roedel in view of Ong to include the above limitations. The motivation to do so would be because an MMM may operate to identify a particular content element with a higher confidence score as compared to a single modal ML model (Devarao: [0025]).
As per claim 16, Roedel in view of Ong and Devarao teaches:
The method of claim 15, wherein the first portion of detected confidential content is associated with the first content type (Devarao: [0025]: CACP may leverage a machine learning (“ML”) model that operates as a multimodal model (“MMM”) that inputs not only images of screen usage. [0026] CACP may operate by altering a video stream (first content type) such that certain information is no longer viewable. Altering of the video stream may include smearing, smudging, blurring, removing, deemphasizing, or obscuring the video stream in whole or in part) and the second portion of detected confidential content is associated with a second content type, and the second content type is distinct from the first content type (Devarao: [0025]: CACP may leverage a machine learning (“ML”) model that operates as a multimodal model (“MMM”) that inputs not only images of screen usage, but also text input (second input type) from a user. [0027]: CACP may alter the background window by replacing the first content element with new content that includes text that is also partially obscured by the active application window. [0073]: Continuing the fifth example, if a user types into the command terminal application a curl command, the processing unit 440 may identify the textual modalities 452-3 and may intercept and alter the display of the terminal window, such that the entire terminal window is unreadable for the target devices 430).
The examiner provides the same rationale to combine prior arts Roedel in view of Ong and Devarao as in claim 15 above.
As per claim 17, Roedel in view of Ong and Devarao teaches:
The method of claim 16, wherein automatically modifying the first portion of detected confidential content is performed by a first ML model of the multimodal ML model, and wherein automatically modifying the second portion of detected confidential content is performed by a second ML model of the multimodal ML model (Devarao: [0073] In some embodiments, the processing unit 440 may be configured to monitor and identify content elements based on a combination of factors across modalities. In a fifth example, if a command terminal application is opened, a first modality that is an application modality 452-2 may be monitored by the processing unit 440. The processing unit 440 may determine that potentially sensitive or unwanted to share content elements are potentially or likely to be displayed based on the monitoring of the application modality 452-2. Continuing the fifth example, if a user types into the command terminal application a curl command, the processing unit 440 may identify the textual modalities 452-3 and may intercept and alter the display of the terminal window, such that the entire terminal window is unreadable for the target devices 430).
The examiner provides the same rationale to combine prior arts Roedel in view of Ong and Devarao as in claim 15 above.
As per claim 18, Roedel in view of Ong and Devarao teaches:
The method of claim 16, wherein a first modification protocol is applied by the multimodal ML model to automatically modify the first portion of detected confidential content (Devarao: [0026] CACP may operate by altering a video stream such that certain information is no longer viewable. Altering of the video stream may include smearing, smudging, blurring, removing, deemphasizing, or obscuring the video stream in whole or in part. Altering of the video stream may include otherwise altering the video stream such that the content elements that are identified are not discernable by a user of a target of the stream (alternatively, viewer)), and wherein a second modification protocol is applied by the multimodal ML model to automatically modify the second portion of detected confidential content (Devarao: [0073]: Continuing the fifth example, if a user types into the command terminal application a curl command, the processing unit 440 may identify the textual modalities 452-3 and may intercept and alter the display of the terminal window, such that the entire terminal window is unreadable for the target devices 430).
The examiner provides the same rationale to combine prior arts Roedel in view of Ong and Devarao as in claim 15 above.
As per claim 19, Roedel teaches:
A method of preventing disclosure of confidential content, comprising:
receiving a conference input of a conferencing session having a plurality of participants, wherein the conference input represents a content type (Roedel: [0078] Where the communication session involves video (with or without audio), the video may be stored as content 218. The content 218 may also include a segment or portion of a live video stream being communicated by the communication client 212);
determining a first user confidentiality level of a first participant of the plurality of participants (Roedel: [0091]. [0093]: To determine the security clearances of the user accounts viewing or participating in the communication session, the obfuscation module 216 may query the communication session server 116 to provide this information. The obfuscation module 216 may query the communication session server 116 at predetermined instances during the communication session. [0094] In response to the query from the obfuscation module 216, the communication session server 116 may reference a user accounts database 326 (illustrated in FIG. 3) to obtain the security clearance value associated with a particular user account (e.g., using a username or other user identifier to query the user accounts database 326) (first participant). The communication session server 116 then communicates the obtained security clearance values to the obfuscation module 216);
identifying, based on the content type, confidential content by evaluating the conference input using a trained (Roedel: [0006]: Training the obfuscation model may include training the obfuscation model to recognize the words and/or images that appear in the uploaded electronic files. In this embodiment, the uploaded electronic files are considered confidential to the organization. [0009]: The obfuscation model may then be used to detect whether an object appearing in an image or video frame is likely to be a confidential object and, further still, which level of confidentiality likely applies to the detected object. [0013]: As the computing device is processing the live video stream, the computing device may employ an image recognition algorithm on the video frames of the live video stream to determine whether the video frames of the live video stream contain an object to be obfuscated. In applying the image recognition algorithm, the computing device may leverage the obfuscation model);
determining a content confidentiality level of the detected confidential content (Roedel: [0009]: The obfuscation model may then be used to detect whether an object appearing in an image or video frame is likely to be a confidential object and, further still, which level of confidentiality likely applies to the detected object. [0091]: The obfuscation module 216 may determine the security clearance required to view the detected object by referencing the obfuscation model 224 and obtaining a likely confidentiality attribute value associated with the detected object. Thus, through a machine-learning algorithm, the obfuscation model 224 also outputs a matrix or other data structure that lists the confidentiality attribute values potentially associated with the detected object, and the likelihood that the detected object should be classified with a particular confidentiality attribute value);
generating, based on the first user confidentiality level of the first participant and the content confidentiality level of the confidential content, a modified conference output by automatically modifying the detected confidential content in the conference input, according to the content confidentiality level (Roedel: [0091]: the obfuscation module 216 may be configured to selectively obfuscate the objects appearing within the content 218 based on one or more user account(s) present in the communication session and in communication with the client device 104. The obfuscation module 216 may be configured to determine a security clearance or authorization clearance required to view the detected object, and then determine which of the user account(s) present in the communication session have the requisite security clearance to view the detected object. The obfuscation module 216 may determine the security clearance required to view the detected object by referencing the obfuscation model 224 and obtaining a likely confidentiality attribute value associated with the detected object. [0095] During the transmission of the communication session, the obfuscation module 216 then compares the received security clearance values with the likely security clearance determined via the obfuscation module 216. The obfuscation module 216 may then manipulate or modify the duplicated video frame(s) to create the obfuscated image(s) 220. Those user account(s) that are not authorized to view the detected object without obfuscation can view the content 218 with the detected object being obfuscated); and
broadcasting the modified conference output as a part of the conferencing session to a second participant, wherein the second participant corresponds to a second user confidentiality level, and the second user confidentiality level is lower than the content confidentiality level (Roedel: [0079]: the obfuscation module 216 determines whether each video frame includes a feature indicating that an object is present in the video frame. [0095]: During the transmission of the communication session, the obfuscation module 216 then compares the received security clearance values with the likely security clearance determined via the obfuscation module 216. Where a security clearance value meets or exceeds the likely security clearance, the obfuscation module 216 records that the corresponding user account is authorized to view the object in the content 218. The obfuscation module 216 may then manipulate or modify the duplicated video frame(s) to create the obfuscated image(s) 220. When the content 218 (or the portion of the content 218 that is buffered in the memory) is transmitted to one or more of the client devices 106-114, the client device 104 may transmit a video where the object appears obfuscated in one or more of the video frame(s) and a video where the object does not appear obfuscated in one or more of the video frame(s), i.e., a second video frame comprising an object to be obfuscated is evaluated based on security clearance of a user account and the confidentiality attribute associated with the object and the object is obfuscated only for a second user account that does not have the requisite security clearance to view the object).
Roedel teaches a machine learning model to detect confidential content but does not teach: using a trained multimodal machine learning (ML) model and a semantic memory, the semantic memory comprises a context of a previous indication of previously detected confidential content, and the confidential content is based on the context as recalled from the semantic memory. However, Ong teaches:
using a trained machine learning (ML) model and a semantic memory, the semantic memory comprises a context of a previous indication of previously detected confidential content, and the confidential content is based on the context as recalled from the semantic memory (Ong: page 403: right column: 2) Bidirectional LSTM Networks: To better utilize the context of a token, we leverage the temporal dependencies among tokens in both directions (forward and backward) through a bidirectional LSTM network [22]. We use both the past features (via forward states) and future features (via backward states) for a specific sequence window. The bidirectional LSTM network is illustrated in Figure 5. As we can see, the contextual information of a word can be leveraged through the learning in the hidden layer which consists of both forward and backward passes. Therefore, the sensitivity classification of a token does not only depend on the features of that token, but is also conditioned on other words present within the context. Page 404: right column: C. Sentence Level In our sentence level experiment, we have trained three different models: BOW, LSTM, and CNN. For this experiment, we manually extract sentences from the documents and label them. To make sure there is sufficient contextual information in each sentence, we only consider sentences which have at least five tokens. Page 405: left and right columns: The word embedding feature vectors are provided as input to the LSTM network with 64 output dimensions. We run 5 epochs for training both LSTM and CNN. D. Token Level: In the token level experiment, we have trained two models: the standard many-to-many LSTM and Bidirectional LSTM (Bi-LSTM). Our dataset contains a total of 9,427 tokens, with only 390 marked as sensitive. The size of word embedding in both LSTM and Bi-LSTM is set to be 128. The embedding feature vectors are then provided as input to the LSTM and Bi-LSTM networks with 64 output dimensions. We use the same setting as the LSTM in sentence level detection. By utilizing the contextual information of target tokens through both forward and backward passes, Bi-LSTM has shown superior overall performance among all the methods, i.e., the context of sensitive information is learned and stored during training. Page 406: left column: VII. CONCLUSION: In this paper, we have introduced several model based methods for detecting sensitive content in the hybrid cloud by leveraging the contextual semantic information through advanced deep learning models. The developed data loss prevention system can detect sensitive content at various degrees of granularity within the document, and immediately respond to any behavior or events violating data sensitivity compliance, i.e., sensitive content is identified based on the context that is recalled from the LSTM network).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to employ the teachings of Ong in the invention of Roedel to include the above limitations. The motivation to do so would be to provide real-time sensitivity detection at various hierarchical degrees of granularity (Ong: pag4 400: left column: paragraph 2).
Roedel in view of Ong does not teach: a trained multimodal machine learning (ML) model. However, Devarao teaches:
a trained multimodal machine learning (ML) model (Devarao: [0023]: CACP may operate by detecting the presence of a screen sharing session and may monitor for and identify specific content elements in a video stream. The monitoring of content elements may be based on performing artificial intelligence operations on the video stream to identify the specific content elements. The specific content elements may be imagery or text that should not be shared or is unintentionally shared by a sharer. For example, the content elements may include private (confidential) information such as usernames, passwords, account numbers, or other information. [0025]: CACP may leverage a machine learning (“ML”) model that operates as a multimodal model (“MMM”) that inputs not only images of screen usage, but also text input from a user, and application statuses of an active computer system. The multimodal model may be configured to identify content that should not be shared as part of the screen sharing session with more accuracy. [0026] CACP may operate by altering a video stream such that certain information is no longer viewable. [0077] In some embodiments, features of the objects may be determined using a supervised machine learning model built using training data).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to employ the teachings of Devarao in the invention of Roedel in view of Ong to include the above limitations. The motivation to do so would be because an MMM may operate to identify a particular content element with a higher confidence score as compared to a single modal ML model (Devarao: [0025]).
As per claim 20, Roedel in view of Ong and Devarao teaches:
The method of claim 19, wherein the multimodal ML model evaluates the conference input based on the content type to detect the confidential content (Devarao: [0025]: CACP may leverage a machine learning (“ML”) model that operates as a multimodal model (“MMM”) that inputs not only images of screen usage, but also text input from a user, and application statuses of an active computer system. The multimodal model may be configured to identify content that should not be shared as part of the screen sharing session with more accuracy. [0026] CACP may operate by altering a video stream such that certain information is no longer viewable. Altering of the video stream may include smearing, smudging, blurring, removing, deemphasizing, or obscuring the video stream in whole or in part. [0073]: Continuing the fifth example, if a user types into the command terminal application a curl command, the processing unit 440 may identify the textual modalities 452-3 and may intercept and alter the display of the terminal window, such that the entire terminal window is unreadable for the target devices 430).
The examiner provides the same rationale to combine prior arts Roedel in view of Ong and Devarao as in claim 19 above.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Roedel in view of Ong and Devarao as applied to claim 1 above, and further in view of prior art of record US 20210012026 to Taylor et al (hereinafter Taylor).
As per claim 4, Roedel in view of Ong and Devarao does not teach the limitations of claim 4. However, Taylor teaches:
wherein the first content type is an audio content type, and wherein modifying the first conference input comprises obscuring audio data associated with the detected first confidential content (Taylor: [0022]: As shown, the machine learning model 200 may receive input in the form of digital media content, e.g., audio, video, and/or images 204, and may determine whether the content contains PII. The machine learning model 200, based on the determination that the content includes PII, may tokenize the PII and output a tokenized version 206 of the input digital media content. [0025]: Upon determining that digital media content contains PII, the machine learning model 200 may replace one or more portions of the PII with various non-sensitive information as tokens. For instance, the list of sample non-sensitive information 250 may include at least static noise 254, white noise 256, silence 258).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to employ the teachings of Taylor in the invention of Roedel in view of Ong and Devarao to include the above limitations. The motivation to do so would be to provide universal employee access of the digital audio and/or video recordings of customer information without violating set compliance procedures or revealing any private or personal customer information (Taylor: [0003]).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Roedel in view of Ong and Devarao as applied to claim 1 above, and further in view of prior art of record US 20200372140 to Jaber et al (hereinafter Jaber).
As per claim 11, Roedel in view of Ong and Devarao does not teach the limitations of claim 11. However, Jaber teaches:
wherein the user confidentiality level of each participant is indicated in an invitation to the conferencing session (Jaber: [0029]: Thus, conference calls can include exchange of as any combination of voice data, video data, text data, image data (e.g., presentation data), file data, or any other types of data. [0051] In the example of FIG. 3, as the organizer 302—represented in the participants selection section 322 as a first participant 306 (“Christie Cline”)—adds or selects individuals to invite to the meeting, she is also able to designate or identify an access level or type for each individual. [0052]: In other words, the organizer can customize the various access levels for each user that she feels would be of utility to her meeting. In this example, the organizer 302 has identified a plurality of invitees 360, including a first participant 362 (“Jason Matterson”), a second participant 364 (“Carl Carson”), and a third participant 366 (“Ben Martins”). At the time each participant was identified for adding to the invitees list (e.g., via a drop-down menu or address book), the system allowed the organizer 302 to specifically select a type/level of access grant 370 desired for each participant. [0053] In FIG. 3, the first participant 362 has been assigned a first access designation 372, the second participant has been assigned a second access designation 374, and the third participant 366 has been assigned a third access designation 376. Each designation can … (b) the data category, sites, or stores that are specifically targeted by the access and that the access will be limited to. [0055] Once the organizer 302 has finalized the meeting invitation, she may submit the request).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to employ the teachings of Jaber in the invention of Roedel in view of Ong and Devarao to include the above limitations. The motivation to do so would be to be helpful to the participants to receive information or guidance regarding the type and level of access that the organizer believes is required or best suited for this meeting (Jaber: [0053]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MADHURI R HERZOG whose telephone number is (571)270-3359. The examiner can normally be reached 8:30AM-4:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Taghi Arani can be reached at (571)272-3787. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MADHURI R. HERZOG
Primary Examiner
Art Unit 2438
/MADHURI R HERZOG/Primary Examiner, Art Unit 2438
Prosecution Insights