DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/01/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
Paragraph [0015] GPT-4.® Remove the dot between the trademark symbol and the preceding character.
Paragraph [0014] recites “such a server”. This should be corrected to “such as a server”
Paragraph [0015] recites “creations of RD session”. They should be corrected to “creation of RD session”
Paragraph [0030] uses reference character 110 for Client Device. However, Figure 4 shows Client Device is 140. The written description should be amended so those reference characters are used consistently with the original disclosure and drawings.
Correct these informalities consistently throughout the specification and drawings. No amendment may introduce new matter.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more Claims particularly pointing out and distinctly Claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more Claims particularly pointing out and distinctly Claiming the subject matter which the applicant regards as his invention.
Claims 3 and 10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 3 and 10 are rejected, the phrase " include one of: accessing a prohibited website and installing or using a prohibited application" renders the claim indefinite because
Claims 3 and 10 recite that the first user actions “include one of: accessing a prohibited website and installing or using a prohibited application.”
The scope of Claim 3 and 10 is not clear because the recited alternative limitations do not distinctly establish the required combination of user actions and corresponding detection indicators.
The applicant is required to clarify the intended alternatives and their relationships to the recited keywords or phrases.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a Claimed invention may not be obtained, notwithstanding that the Claimed invention is not identically disclosed as set forth in section 102, if the differences between the Claimed invention and the prior art are such that the Claimed invention as a whole would have been obvious before the effective filing date of the Claimed invention to a person having ordinary skill in the art to which the Claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the Claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application is currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 6, 8, 9, 13, 15, 16, and 19 are rejected under 35 U.S.C. § 103 as being unpatentable over Fielding (US 2013/ 0138765A1), in view of Spencer (US 2024/0406280A1)
Regarding Claim 1, Fielding teaches a method of auditing user actions performed in remote desktop (RD) sessions, the method comprising (see e.g., para. [0007], “Methods, media, and systems for monitoring access to computer environments … setting up a remote desktop access session” and see e.g., para. [0052], Fig 6, “review/audit a session” on user action, and see e.g., para. [0048], “determine if the session should be flagged”) acquiring a first video file that visually captures a plurality of first user actions (see e.g., para. [0035], Fig 4, “process 410 can next request and subsequently receive a session to be encoded from cache agent process 420” and see e.g., para. [0042], Fig. 4., “The output file can then be transcoded into ... suitable video encoding formats” and see e.g. para. [0037], Fig. 4., “process 410 can get a screen update and update the frame buffer based on the screen update. The frame buffer can then be written to an output file in a target format” and see e.g., para. [0031], Fig. 3., “processes can be received, forwarded, and recorded…any suitable screen updates, keystrokes, and/or mouse inputs (that equate to user actions on the plurality of devices) can be recorded”) that were performed in a first RD session by a remote device hosting the first RD session (see e.g., para. [0030], Fig. 3, “Once the user has logged on, processes 310 and 330 can setup a remote desktop access session at 315 and 336, and process 320 can receive, forward, and record remote desktop access setup messages” and see e.g., para. [0024], Fig. 2, “Capture agent virtual machines 204. Capture agent 202 and virtual machines 204 can be used to provide connections between technician workstations 102 and computer environments 104 and monitor those connections”) in response to instructions from a client device of the first RD session (see e.g., para. [0027], Fig. 3, “a user (client device) request to access a given computer environment (Remote device) at 312” and see e.g., para [0024], Fig 2, “remote desktop access servers (RD session) can connect to computer environments 104” and see e.g., para. [0020], “keystroke inputs, mouse inputs sent between the technician workstation and the computer environment (instructions from a client device of the RD session) … automatically monitored for alert conditions based on keystroke inputs, mouse inputs”) capturing user actions in RD sessions (see e.g. para. [0031], Fig. 3, “keystrokes, and mouse inputs sent between these processes can be received, forwarded, and recorded, and these screen updates and keystrokes sent to OCR agent 218 by process 320”) searching the first text file for keywords or phrases that have been identified as being associated with prohibited or suspicious actions (see e.g., para. [0052], “Any suitable search parameters can be used… a search can return a list of keyword occurrences” and see e.g., para. [0048], Fig. 5, “process 520 can apply one or more defined rule(s) to the text, supporting data, keystrokes, clipboard content, files…a rule may define that certain action (e.g., such as generating an alert, flagging a session /prohibited or suspicious actions/, terminating a session, pausing a session, etc.) should be taken when certain text…is present on the remote desktop”) in response to detecting one of the keywords or phrases in the first text file, terminating the first RD session (see e.g., para. [0048], Fig. 5, “certain action (e.g., such as generating an alert, flagging a session, terminating a session, pausing a session, etc.) should be taken when certain text…is present…”).
However, Fielding does not disclose generating a first text file describing the first user actions from the first video file, by using a generative artificial intelligence (AI) model that has been trained to generate text descriptions of user actions from video data.
In the same field of endeavor, Spencer discloses generating a first text file describing the first user actions from the first video file (see e.g., para. [0016], “disclosure relate to generating a user activity” and see e.g., para. [0017], Fig. 2A, “any number of implicit and/or explicit signals may be processed …including, but not limited to, a screen recording (e.g., including a video and/or a set of screenshots) …”) by using a generative artificial intelligence (AI) model that has been trained to generate text descriptions of user actions from video data (see e.g. para. [0017], “... a screen recording is processed using a generative machine learning model to derive one or more user activity signals …a multimodal generative model is used to process image data of the screen recording and generate a textual and/or programmatic description” see also e.g., para. [0053], “generating a screen recording comprising one or more images and/or frames of video data”).
It would have been obvious to a person of ordinary art, before the effective filling date, to modify Fielding’s RD-session monitoring system by applying Spencer’s generative machine-learning model to Fielding’s recorded remote-desktop screen-data because a generative machine learning model may improve recommendation performance as suggested by Spencer (see e.g., para. [0016]).
Regarding Claim 2, Fielding teaches, wherein the first video file is received at a connection device separate from the remote device (see e.g., para. [0059], Fig. 2, “encoding of the session by encoding agent 226” (Fig. 2) has completed successfully as indicated by ‘video ready,’ whether video transcoding was unsuccessful” and see e.g., para. [0035], Fig. 4, “…this request can be received by cache agent process 420 at 423, after which process 420 can retrieve the session from temporary storage 224 (FIG. 2) at 424, send the session to encoding agent process 410” The encoding agent 226 and the and the computer environment /remote device/ 104 are separate both architecturally and physically, see Fig. 2, only connected through VPN) that monitors RD sessions hosted by the remote device (see e.g., para. [0024], Fig 2, “Capture agent 202 and virtual machines 204 can be used to provide connections between technician workstations 102 and computer environments 104 and monitor those connections”) terminating the first RD session comprises transmitting a request from the connection device to the remote device to terminate the first RD session (see e.g., para. [0033], Fig. 3, “When the remote desktop access session is to be terminated, process 310 can receive a user command to end the remote desktop access session at 317 …Capture agent process 320 can then receive and forward to virtual machine process 330 the end session message at 328 and process 330 can receive this message at 338”).
Regarding Claim 6, acquiring a second video file that visually captures a plurality of second user actions that were performed in a second RD session (the claim 1 acquisition approach would be repeated for another session) generating a second text file describing the second user actions from the second video file, by using the generative AI model (the claim 1 proposed generative description mechanism would process the second recording with the output stored as the second text file) detecting one of the keywords or phrases in the second text file;” (the claim 1 detection of keyword or phrase on the RD session would be repeated on the second file) in response to detecting the one of the keywords or phrases in the second text file (the claim 1 action to be taken due to detection of keywords or phrases could be related on the second file) alerting an administrator of the second RD session to review the second video file (Fielding teaches security notification (see e.g., para. [0048], Fig. 5, “The process can then determine if an alert to security should be generated at 527, and if so, alert security at 528” and see e.g., para. [0050], “a signal can be sent to the alert agent so that an alert is raised that can lead to session termination, flagging of the session for the review, authorization request to security officer to continue”).
Regarding Claim 8, Fielding discloses a non-transitory computer-readable medium comprising instructions that are executable in a computer system, wherein the instructions when executed cause the computer system to carry out … (see e.g., para. [0064], “any suitable computer readable media can be used for storing instructions for performing the processes described … non-transitory computer readable media.” See also claim 9 of the reference.) The remaining limitations of claim 8 correspond substantively to claim 1 and are rejected for the same reasons set forth above regarding claim 1.
.
Regarding Claim 15, Fielding discloses a computer includes a processor and memory, wherein the computer is configured to use the processor to execute instructions from the memory (see e.g., para. [0063], “a computer or a special purpose device…include any suitable components such as one or more hardware processor (each of which can be a microprocessor, digital signal processor, a controller, etc.), memory, communication interfaces, display controllers, input devices, etc.” and see e.g., para. [0064], “any suitable computer readable media can be used for storing instructions for performing the processes described…”). The remaining functional limitation of claim 15 correspond substantially to the method limitations of claim 1 and are rejected for the reasons set forth above regarding claim 1.
Claim 9 is substantially similar to claim 2 except that it is a system claim and rejected for the same reasons as given in the rejection of claim 2.
Claims 13 and 19 are substantially similar to claim 6 except that they are device claims and are rejected for the same reasons as given in the rejection of claim 6.
Claim 16 is substantially similar to claim 2 except that it is a device claim and it is rejected for the same reasons as given in the rejection of claim 2.
Claims 3 and 10 are rejected under 35 U.S.C. § 103 as being unpatentable over Fielding, in view of Spencer, and further in view of Venugopal (WO2023097245A1)
Regarding Claim 3, Feilding teaches wherein the first user actions include one of: accessing a prohibited … (see e.g., para. [0020], “unauthorized activity of a user/technician at the technician workstation”) and wherein the keywords or phrases include one of: a name (see e.g., para. [0045], Fig. 2, “This text, along with the keystrokes received at the screen updates, can then be provided to alert agent 220 (FIG. 2) that can determine if an action should be taken based on the text on the remote desktop and corresponding keystrokes” and see e.g., para. [0048], “a rule may define that certain action .. should be taken when certain text (e.g., such as “confidential,” account numbers, credit card numbers, social security numbers, passwords, personal identification numbers (pins), etc.) is present on the remote desktop”).
However, Feilding does not disclose about monitoring prohibited website and installing or using a prohibited application… or uniform resource locator (URL) of the prohibited website and a name of the prohibited application.
In the same field of endeavor, Venugopal teaches see e.g., para. [0060] “determine whether the first application is associated with either the list of … prohibited computer applications” and see e.g., para. [0059], Fig. 9, “a computer-implemented application monitoring method 900 including executable instructions 207 (Fig. 2) programmed for execution in a computing environment for controlling the usage of one or more computer applications or websites … computing device located in a home or office work environment”).
A person of ordinary skill in the art would have been motivated, before the effective filing date of the claimed invention to incorporate Venugopal’s prohibited application monitoring into Fielding’s remote desktop (RD) monitoring system to detect unauthorized application use during monitored session as suggested by Venugopal para. [0058].
Claim 10 is substantially similar to claim 3 except that it is a device claim and it is rejected for the same reasons as given in the rejection of claim 3.
Claims 4, 11, 17 are rejected under 35 U.S.C. § 103 as being unpatentable over Fielding, in view of Spencer, and further in view of Li (Li et.al., CVPR 2020)
Regarding Claim 4 Spencer discloses training the generative AI model (see e.g., para. [0041], Fig. 2A, “machine learning model is trained”) capturing user actions performed in RD sessions (see e.g., para. [0028], Fig. 1, “signal processor 118 captures user activity signals according to aspects described herein (e.g., in response to initiating a feedback session) …user signal processor 118 processes a screen recording to derive one or more user activity signals therefrom” and see e.g., para. [0045], Fig.1, “the set of user signals is received from a remote computing device”).
However, Spencer does not explicitly disclose video files each visually captured and with a plurality of text descriptions corresponding to the plurality of video files
In the same field of endeavor, Li et al., teaches multiple annotated software tutorial video clips (see e.g., p. 12523 “video screen capture” see e.g., sec. 4.0, p. 12526, “loads each video …as temporal segments …” and see e.g., sec. 5.0, p. 12528, “to generate text description for a given tutorial video clip, we build upon the sequence-to-sequence model…that accommodates both input frame sequence and output word sequence of variable lengths.” and see Li, sec. 6.3, p. 12529, “generate a text description of the major user actions in this clip” and see e.g., sec. 3.0, p. 12525 “With the collected dataset, we build machine learning models”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Spencer’s training the generative model using multiple screen-recording video examples paired with the corresponding textual descriptions, as taught by Li. The teachings provide a concrete training approach for generative processing of screen recording into textual description.
Claim 11 and Claim 17 are substantially similar to claim 4 except they are device claims and are rejected for the same reasons as given in the rejection of claim 4.
Claims 5, 12, 18 are rejected under 35 U.S.C. § 103 as being unpatentable over Fielding, in view of Spencer, and further in view of Kihara (US 2019/0188398 A1)
Regarding Claim 5, acquiring a second video file that visually captures a plurality of second user actions that were performed in a second RD session; (the acquisition approach proposed for claim 1 could be repeated for another monitored session. Fielding expressly accommodated multiple session see e.g., para [0057], “simultaneous encoding of multiple sessions”) generating a second text file describing the second user actions from the second video file, by using the generative AI model (the proposed claim 1 description mechanism would likewise process the second recording) detecting one of the keywords or phrases in the second text file (the proposed claim 1 security-rule would be repeated for the second text file) in response to detecting the one of the keywords or phrases in the second text file, (see the presence of the specific text causing an action in claim 1 could be repeated).
Kihara discloses causing a warning message to be displayed in a graphical user interface (GUI) of the second RD session (see e.g., para. [0074], Fig. 8, “the session management unit 13 causes the display unit 31 of the client device 30 to display the termination alert (S206)”) wherein the warning message describes one of the plurality of second actions associated with the detected one of the keywords or phrases in the second text file (see e.g., para. [0075], Fig. 10, “the termination alert includes a message of “Remote desktop connection will be terminated in 20 seconds”, and “Current status” and “To avoid termination” are indicated [on the GUI] in addition to the message. The field “Current status” indicates the current location of the user and a name of a file used by the user which is prohibited at the current location”).
It would have been obvious to an ordinary person of skill in the art before the effective filing date of the claimed invention to use Kihara’s activity-specific warning functionality into the proposed Fielding and Spencer combination system to display, within the RD session GUI, a warning identifying the detected prohibited user of the specified activity that triggered the warning as suggested by Kihara (see Fig. 11).
Claims 12 and Claim 18 are substantially similar to claim 5 except they are device claims, and both are rejected for the same reasons as given in the rejection of claim 5.
Claims 7, 14, 20 are rejected under 35 U.S.C. § 103 as being unpatentable over Fielding, in view of Spencer, and further in view of Russell (US 2005/0271251A1)
Regarding Claim 7, acquiring a second video file that visually captures a plurality of second user actions that were performed in a second RD session (the video-acquisition mechanism applied to claim 1 would likewise acquire a video file for another monitored RD session) generating a second text file describing the second user actions from the second video file, by using the generative AI model (the same generative AI processing applied to the first video file in claim 1 would be applied to the second video file to generate a corresponding description of the second user actions) searching the second text file for the keywords or phrases; and in response to determining (the keyword-or-phrase search applied to the first text file in claim 1 would likewise applied to the second text file).
Russell discloses that none of the keywords or phrases are in the second text file, deleting the second video file from memory or storage of a computer that acquired the second video file. (See e.g., para. [0072] “prioritize the storage of data according to relevancy and importance… delete the video once it has been confirmed that no activity took place. If, however, a potential security event is detected in the video or by an external sensor or system, the system can be configured to store video data in uncompressed format continuously until it is determined that the video data no longer provides any value” and see e.g., para. [0091] “ Data Storage Management Rules include Data Importance Assessment Rules (“Assessment Rules”) that specify … one or more conditions in which the importance of currently stored data will automatically be assessed…, all stored video data that has not been linked to an event is deleted, and all stored video data associated with positively identified authorized personnel is compressed and archived”).
It would have been obvious to a person of ordinary skill in the art, before the effective filling date of the claimed invention to incorporate Russell’s rule-based video -retention policy into the Fielding RD-session auding system for intelligent automatic reduction of stored data such that video data of most interest are maintained while less important data is deleted as suggested by Russell (see e.g., Abstract).
Claim 14 and Claim 20 are substantially similar to claim 7. Both claims are rejected for the same reasons as given in the rejection of claim 7.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIBEBU HAILEMARIAM whose telephone number is (571)272-0221. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIBEBU HAILEMARIAM/Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125