DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This Office Action is in response to the application filed on 05/06/2024. Claims 1-20 have been examined.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/06/2024 filed in accordance with the provisions of 37 CFR 1.97. Accordingly, it is being considered by the examiner.
Claim Objection
Claim 6 is objected to under 37 CFR 1.75(c) because it does not appear to further limit the subject matter of claim 5. Claim 6 recites that "the second video is of the same surgical procedure, or the second video is of a different surgical procedure." These alternatives collectively encompass all possible relationships between the second video and the referenced surgical procedure and therefore do not impose a meaningful additional limitation on claim 5. Applicant is requested to amend the claim to further limit the subject matter of the parent claim or otherwise explain how the claim further limits claim 5.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 6 is unclear because the phrase "same surgical procedure" lacks a reference point. The claim fails to specify what procedure the second video is being compared against. Consequently, the metes and bounds of the claim cannot be determined with reasonable certainty.
Claim 6 recites "wherein the second video is of the same surgical procedure, or the second video is of a different surgical procedure." In light of the specification, the Examiner interprets the phrase "same surgical procedure" as referring to the surgical procedure associated with the first video recited in claim 1 and the second video recited in claim 5. See, e.g., Spec. [0018]. However, claim 6 fails to expressly identify the reference procedure against which the second video is being compared. As drafted, the metes and bounds of the claim are unclear because the phrase "same surgical procedure" lacks an explicit antecedent basis or reference point within the claim language itself. Accordingly, claim 6 fails to particularly point out and distinctly claim the subject matter which Applicant regards as the invention.
Double Patenting
The Obviousness double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the "right to exclude" granted by a patent and to prevent possible harassment by multiple assignees. A non-statutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Langi, 759 F.2d 887,225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937,214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CPR l.32l(c) or l.32l(d) may
be used to overcome an actual or provisional rejection based on non-statutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717 .02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CPR l.32l(b).
The filing of a terminal disclaimer by itself is not a complete reply to a non-statutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B. l. For a reply to a non-final Office action, see 37 CPR 1.11 l(a). For a reply to final Office action, see 37 CPR 1.113(c). A request for reconsideration while not provided for in 37 CPR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13. The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto processed and approved immediately upon submission. For more information about eTerminal Disclaimers, to refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of non-statutory double patenting as being unpatented over claims 1-20 of U.S, Application No. 18/707,707.
Although the claims of the instant application are not identical, to those of the referenced patent, they are not patentably distinct therefrom. The difference between the claim sets are merely in terminology, as illustrated in the claim comparison table below, and do not result in any patentable distinction.
Accordingly, the instant claims are considered to be an obvious variation of the claims of the cited patent.
To overcome this rejection, a terminal disclaimer must be filed. The terminal disclaimer must disclaim any term of the instant application that would extends beyond the term of the referenced patent and must include the required common ownership and enforceable provisions under C.F.R. § 1.321.
The following table provides an exemplary comparison between representative claim, and shows that the difference are merely in wording and do not constitute a patentable distinction.
18/707,701
(Instant Application)
18/707, 707
(US-20240428956-A1 )
EXEMPLARY CLAIM 1
CLAIM 1
1. A computer-implemented method comprising:
receiving, by a processor, a first video portion from a video of a surgical procedure, the video comprising a sequence of video portions;
generating, by the processor, a first latent representation of the first video portion using an encoder machine learning model;
comparing, by the processor, the first latent representation with a plurality of latent representations representing previously analyzed video portions;
in response to the first latent representation being within a predetermined threshold of a second latent representation from the plurality of latent representations, storing, by the processor, in a compressed video corresponding to the video of the surgical procedure, the second latent representation; and
outputting, by the processor, the compressed video, which is a sequence of latent representations respectively corresponding to the sequence of video portions.
2. The computer-implemented method of claim I, wherein the video is being transmitted to the processor as the surgical procedure is being performed.
1. A computer-implemented method comprising:
receiving, by a processor, a first video portion from a video of a surgical procedure, the video comprising a sequence of video portions;
generating, by the processor, a first latent representation of the first video portion using an encoder machine learning model;
comparing, by the processor, the first latent representation with a plurality of latent representations representing previously analyzed video portions, the comparing comprising generating and executing a query that includes the first latent representation as a search parameter;
in response to the first latent representation being within a predetermined threshold of a second latent representation from the plurality of latent representations, retrieving, by the processor, from the previously analyzed video portions, a second video portion corresponding to the second latent representation; and
outputting, by the processor, the second video portion as a candidate for playback.
2. The computer-implemented method of claim I, wherein the video is being transmitted to the processor as the surgical procedure is being performed.
Claims 3-20 list all the same elements of claims 3-20. Therefore, the supporting rationale of the rejection applies equally as well to claims 3-20.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C.
102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction
of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status
The following is a quotation of 35 U.S.C. 103 which forms the basis for all
obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention
is not identically disclosed as set forth in section 102, if the differences between the claimed
invention and the prior art are such that the claimed invention as a whole would have been obvious
before the effective filing date of the claimed invention to a person having ordinary skill in the art
to which the claimed invention pertains. Patentability shall not be negated by the manner in which
the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35
U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness
Claims 1-3, 16-17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Khalid et al (US 2020/0367974) in view of Miao et al. (US 2019/0272415 A1) hereinafter “MIAO”.
Regarding Claim 1 Khalid-Miao
KHALID discloses 1. A computer-implemented method comprising:
a) receiving, by a processor, a first video portion from a video of a surgical procedure, the video comprising a sequence of video portions; (Khalid, FIG. 2A, Input frame 250, FIG. 3, 302 “Receive surgical procedure video data object 302”, Claim 1 "receive a first data set comprising one or more surgical procedure video data sets comprising a plurality of frames" (¶201). "Receive surgical procedure video data object 302" (Fig. 3; ¶[0201]).
The claimed "first video portion" reads on a frame or portion of the surgical procedure video data set received by Khalid. The plurality of frames in the surgical procedure video corresponds to the claimed sequence of video portions.
The claimed first latent representation reads directly on Khalid's embedding 212.
Claim 1 “receive a first data set comprising one or more surgical procedure video data sets comprising a plurality of frames, the plurality of frames comprising at least one surgical instrument, and label data for each of the one or more surgical procedure video data sets representative of surgical performance”);
b) generating, by the processor, a first latent representation of the first video portion using an encoder machine learning model (Khalid, [0146] "The dimensionality reduction architecture 122 is comprised of an encoder and a decoder that can be jointly trained to represent data with a much smaller set of dimensions.".
Khalid further discloses: "The resulting output (e.g., the embedding 212) is effectively a latent representation (i.e. a compressed frame) of the input data, or alternatively referred to as a compact representation of the frame data." (¶147).
c) . . .
d) . . .
The following teaching by Khalid is Relevant to the above limitation, but does not filter out the way the claim filters out.
Khalid [0230] “In example embodiments, the internal view detector 604, is used to filter out frames that consist of content which is irrelevant to assessing surgical performance, such as an external view of the surgical staff. The internal view detector 604 can help in efficiently allocating computing resources, or increasing speed or accuracy as it can remove irrelevant frames, which may be a feature of the surgical performance video data object (e.g., the camera capturing the laparoscopic procedures will capture video data of it being inserted into and removed from the body multiple times during the surgery”).
e) outputting, by the processor, the compressed video, which is a sequence of latent representations respectively corresponding to the sequence of video portions (Khalid, [0147] “The resulting output (e.g., the embedding 212) is effectively a latent representation (i.e. a compressed frame) of the input data, or alternatively referred to as a compact representation of the frame data.”. [0027] "The trained encoder portion passes the respective frame embeddings (i.e., compressed frames)." The sequence of embeddings generated for video frames corresponds to a compressed representation of the surgical video.).
Khalid does not explicitly disclose the limitations of c and d
c) comparing, by the processor, the first latent representation with a plurality of latent representations representing previously analyzed video portions;
d) in response to the first latent representation being within a predetermined threshold of a second latent representation from the plurality of latent representations, storing, by the processor, in a compressed video corresponding to the video of the surgical procedure, the second latent representation; and
However, MIAO discloses
c) comparing, by the processor, the first latent representation with a plurality of latent representations representing previously analyzed video portions; (MIAO, FIG. 1 STEP 101, “Compare an obtained first facial image with pre-stored facial image information in an image database". (Step 101; [0037])
d) in response to the first latent representation being within a predetermined threshold of a second latent representation from the plurality of latent representations, storing, by the processor, in a compressed video corresponding to the video of the surgical procedure, the second latent representation; (Miao discloses determining similarity between feature information of a current facial image and pre-stored facial image feature information. In [0092]-[0093] Miao teaches that "a set value (configurable, e.g., 0.86) is used as a threshold" and that "similar images are filtered." Miao further teaches that "in the case of the comparison result being higher than the threshold, it is indicated that the face has been uploaded to the cloud server, and it is unnecessary to upload again.". [0033] (similarity comparison), ¶0040 (store/add feature information when new), [0092]-[0093] (threshold), [0096],[0101] (duplicate determination / not storing duplicate).
Therefore, it would have been obvious to a person having ordinary skill in the art , before the effective filing date of the claimed invention, to modify Khalid's method of generating latent representations (embeddings) of surgical-video frames with Miao's similarity-based comparison and storage techniques by comparing a newly generated latent representation to previously stored latent representations, determining whether the newly generated latent representation matches a previously stored latent representation based on a similarity threshold, and selectively storing latent representations based on the comparison result, in order to avoid redundant storage of substantially similar video content, reduce storage requirements, reduce computational burden, and improve the efficiency of processing large surgical-video datasets while preserving representative information for subsequent retrieval and analysis. (Khalid, [0006]-[0009] and Miao, ([0040], [0092]-[0093]).
Regarding Claim 2 Khalid-Miao
Khalid-Miao discloses 2. The computer-implemented method of claim 1, wherein the video is being transmitted to the processor as the surgical procedure is being performed. (Khalid expressly teaches receiving and processing surgical procedure video while the procedure is being performed. Specifically Khalid discloses “A system having the machine learning architecture receives as inputs video data of surgical procedures, and processes them either in real-time or near-real time...” [0018]. Khalid further discloses "the processor 104 stores real-time streamed surgical performance video data as sets of a plurality of frames…" ([0136]).
Regarding Claim 3 Khalid-Miao
Khalid-Miao discloses 3. The computer-implemented method of claim 1, wherein the video is captured using one from a group of cameras comprising an endoscopic camera, a portable camera, and a stationary camera. (Khalid, [0325] “An example camera 30 is a laparoscopic or procedural view camera resident in the surgical unit, ICU, emergency unit or clinical intervention units.” [0230] (e.g., the camera capturing the laparoscopic procedures will capture video data of it being inserted into and removed from the body multiple times during the surgery).).
Regarding Claim 16 Khalid-Maio
Khalid discloses A computer program product comprising a memory (Fig. 1, 108 “Memory”) device having computer-executable instructions stored thereon, which when executed by one or more processor (Fig. 1, 104 “Processor” cause the one or more processors to perform a method to compress a data collection system comprising a plurality of surgical data corresponding to surgical procedures ([0004] a system and method for assessing surgical procedures based on processing surgical procedure video data set with a machine learning architecture. "surgical video frames" [0117] “…the processor 104 executes instructions in memory 108 to …” Fig. 1; ¶ [0118] and [0124].), the method comprising:
a) generating a latent representation space corresponding to the data collection system, the latent representation space comprising a plurality of latent representations, wherein a latent representation is a vector representation of a portion of a surgical data" (Khalid teaches a plurality of latent representations generated from surgical video data, wherein "the embedding 212 is effectively a latent representation (i.e., a compressed frame) of the input data" ([0147]). See also ([0020], [0050]));
b) "in response to receiving a first surgical data, generating a compressed surgical data corresponding to the first surgical data" (Khalid teaches compressing surgical video data using an encoder that generates embeddings from surgical video frames ([0020], [0147]));
i) "generating a first latent representation of a first portion from the first surgical data" (Khalid teaches generating an embedding from a frame of surgical video data, wherein "the embedding 212 is effectively a latent representation (i.e., a compressed frame) of the input data" ([0147])); and
ii) …
iii) … "storing, in the compressed surgical data, the first latent representation in place of the first portion" (Khalid teaches storing the embedding as a compressed representation of the input data, wherein "the embedding 212 is effectively a latent representation (i.e., a compressed frame) of the input data" ([0147])).
Khalid does not explicitly disclose:
ii) "comparing the first latent representation with the latent representations in the latent representation space";
iii) "in response to determining, from the latent representation space, a second latent representation that is similar to the first latent representation, storing, in the compressed surgical data, the second latent representation in place of the first portion"; and
iv) "based on determining that the first latent representation is not similar to any of the plurality of latent representations, adding the first latent representation to the latent representation space, and storing, in the compressed surgical data, the first latent representation in place of the first portion."
However, Miao teaches:
ii) "comparing the first latent representation with the latent representations in the latent representation space" (Miao, by comparing current feature information with previously stored feature information in an image database ([0037]-[0040]);
iii) "in response to determining, from the latent representation space, a second latent representation that is similar to the first latent representation, storing, in the compressed surgical data, the second latent representation in place of the first portion" (Miao, by determining whether current feature information matches previously stored feature information in the image database ([0037]-[0040]); and
iv) "based on determining that the first latent representation is not similar to any of the plurality of latent representations, adding the first latent representation to the latent representation space, and storing, in the compressed surgical data, the first latent representation in place of the first portion" (Miao, by teaching that when no matching stored information is found, "the qualified facial image and/or feature information thereof is added to the image database" ([0040]).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Khalid's latent-representation compression system with Miao's similarity-comparison and database-update techniques in order to avoid storing redundant latent representations and instead utilize previously stored matching representations when similar content is identified, while adding newly generated representations when no sufficiently similar stored representation exists. Miao expressly teaches comparing current feature information with previously stored feature information and adding new feature information when no match is found ([0037]-[0040]). Applying these known redundancy-management techniques to Khalid's latent representations would have predictably reduced redundant storage while preserving the ability to reconstruct the underlying surgical video data. Such modification would have yielded predictable results.
Regarding Claim 17 Khalid-Miao
Khalid-Mao discloses 17. The computer program product of claim 16, wherein, during playback of the compressed surgical data, reconstructing the first portion of the first surgical data based on the first latent representation or the second latent representation stored in the compressed surgical data. (Khalid discloses that a decoder "reconstructs frames from the compressed frames generated by the encoder portion" ([0020]) and recreates the original image "from the low dimensional embedding" ([0148]). See also ([0165]). )
Regarding Claim 20 Khalid-Miao
Khalid-Miao discloses 20. The computer program product of claim 16, wherein the second latent representation is linked to a second portion from a second surgical data. (Khalid teaches a plurality of latent representations generated from surgical video data, wherein "the embedding 212 is effectively a latent representation (i.e., a compressed frame) of the input data" ([0147]). See also ([0020], [0050]), see also [0004]);
Claim Rejections - 35 USC § 103
Claims 4-7, and 18 is rejected under 35 U.S.C. §103 as being unpatentable over KHALID-MIAO and further in view of Yun et al. (US 11,310,539 B1) hereinafter “Yun”.
Regarding Claim 4 Khalid-Maio-Yun
Khalid-Maio-Yun 4. The computer-implemented method of claim 1, wherein, the second latent representation is linked to a second video portion, which is stored in video format. (Khalid, [0147] “..The resulting output (e.g., the embedding 212) is effectively a latent representation (i.e. a compressed frame) of the input data, or alternatively referred to as a compact representation of the frame data.”[0147]). Also Yun discloses: "Embedding data source 138 associates an embedding (generated by embedding generator 136) with the video item from which the embedding was generated." (col. 5, lines 25-27). Yun further discloses: "each of data sources 132-134 includes an embedding (generated by embedding generator 136) that is stored in association with the corresponding video item." (col. 5, lines 32-35). Yun additionally discloses that: "a video item may be associated with multiple embeddings, each generated based on different text portions of the video item." (col. 5, lines 38-41).
Therefore, Yun teaches that embeddings are associated with the source video content from which the embeddings are generated and that multiple embeddings may correspond to different portions of a video item. Therefore, Yun teaches a latent representation linked to video content corresponding to a portion of a video, as claimed.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Khalid-Miao to store the latent representation in association with a separately stored video file as taught by Yun, because maintaining an association between an embedding and the corresponding video content enables efficient retrieval of the video content using the embedding and vice versa, thereby facilitating video matching, indexing, and storage operations while preserving the relationship between the compressed representation and the underlying video data. Such modification would have yielded predictable results.
Note: The motivation that was utilized in the rejection of claim 4, applies equally as well to claims 5-7.
Regarding Claim 5 Khalid-Maio-Yun
Khalid-Maio-Yun discloses 5. The computer-implemented method of claim 4, wherein the second video portion is from a second video. (Khalid, [0147] “The resulting output (e.g., the embedding 212) is effectively a latent representation (i.e. a compressed frame) of the input data, or alternatively referred to as a compact representation of the frame data.”)
Regarding Claim 6 Khalid-Maio-Yun
Khalid-Maio-Yun discloses 6. The computer-implemented method of claim 5, wherein the second video is of the same surgical procedure, or the second video is of a different surgical procedure. (Khalid, [0023] “The subsequent surgical performance video data may be the same surgical performance video data used to train the dimensionality reduction architecture. According to some scenarios, the subsequent surgical performance video data are videos of different procedures ….”Any second video within the plurality of videos would necessarily either depict the same surgical procedure as another video or a different surgical procedure.)
Regarding Claim 7 Khalid-Maio-Yun
Khalid-Miao-Yun discloses 7. The computer-implemented method of claim 4,
wherein the second video portion is stored as a separate video file, and the second latent representation is linked to the separate video file. (Yun's discloses that "Embedding data source 138 associates an embedding (generated by embedding generator 136) with the video item from which the embedding was generated" (col. 5, lines 25-27) and that "each of data sources 132-134 includes an embedding (generated by embedding generator 136) that is stored in association with the corresponding video item" (col. 5, lines 32-35). The claimed "second latent representation" reads on Yun's embedding, while the claimed "linked to the separate video file" reads on Yun's association between the embedding and the corresponding video item. Accordingly, Yun teaches a latent representation that is linked to separately stored video content (col. 5, lines 25-35).
Regarding Claim 18 Khalid-Maio-Yun
Khalid-Maio discloses 18. The computer program product of claim 16, .
Khalid-Miao does not explicitly disclose wherein adding the first latent representation to the latent representation space comprises:
storing the first portion as a separate file; and
linking the first latent representation with the separate file;
However, Yun discloses
storing the first portion as a separate file (Yun, the generated embedding is stored in association with the corresponding video item" (col. 5, lines 25-35).; and
linking the first latent representation with the separate file;
(Yun, associating the embedding with the video item. data sources 132-134 includes an embedding (generated by embedding generator 136) that is stored in association with the corresponding video item.) An embedding is "associated" with the video item from which it was generated and is "stored in association with the corresponding video item" (col. 5, lines 25-35)).
Claim Rejections - 35 USC § 103
Claims 8-10, and 19 is rejected under 35 U.S.C. §103 as being unpatentable over Khalid-Miao further in view of Yokouchi et al. (US 20140347704 A1) hereinafter “Yokouchi ”
Regarding Claim 8 Khalid-Miao-Yokouchi
Khalid-Miao discloses 8. The computer-implemented method of claim 1,
Khalid-Miao does not explicitly disclose wherein the video is deleted after the compressed video is stored.
However, Yokouchi discloses wherein the video is deleted after the compressed video is stored. ("[0068] “The image compressor 136 compresses the input image according to an encoding method such as Joint Photographic Experts Group (JPEG), and thereby generates compressed image data. The image compressor 136 deletes the input image after generating the compressed image data.")
Therefore, it would have been obvious to a person having ordinary skill in the art before
the effective filing date of the application to modify the teachings of Khalid-Miao with Yokouchi to create the system of Khalid-Miao, wherein the video is deleted after the compressed video is stored as suggested by Yokouchi.
One of ordinary skill in the art would have been motivated to incorporate this feature into Khalid-Maio “such that the data amount after compression would be optimal,,,: (Yokouchi.,[0007]
Note: The motivation that was utilized in the rejection of claim 8, applies equally as well to claims 9, 10, and 17
Regarding Claim 9 Khalid-Miao-Yokouchi
Khalid-Miao-Yokouchi discloses 9. The computer-implemented method of claim 8, further comprising:
during playback of the video, reconstructing one or more portions of the video based on the sequence of latent representations stored in the compressed video. (Khalid, the limitation is taught by Khalid's disclosure of reconstructing frames from compressed frame embeddings using a decoder portion of the dimensionality reduction architecture ([0020], [0148], [0165]). Specifically, Khalid teaches reconstructing image frames from low-dimensional embeddings, wherein the embeddings correspond to compressed frame representations. Accordingly, Khalid teaches reconstructing portions of a video from stored latent representations. (¶¶ [0020], [0148], [0165]).
Khalid discloses
[0020] The dimensionality reduction architecture includes, an encoder portion (i.e., the first portion), which compresses the frames, and a decoder portion (i.e., the second portion), which reconstructs frames from the compressed frames generated by the encoder portion.” See also [0120])
[0050] FIG. 5 depicts an image and images reconstructed from embeddings of different dimensions, in accordance with some embodiments;
[0148] Generally, the right portion of the dimensionality reduction architecture 122 is the decoder which applies a series of up-sampling or 2D transpose convolutions to the data in order to recreate the original image (e.g., a reconstructed image 260) from the low dimensional embedding.
[0165] The decoder portion of the dimensionality reduction architecture 122 generates the reconstructed representation 260 for each frame 250 of the surgical performance video data object, wherein the further partially decompressed frame of the respective frame 250, generated by the fifth convolutional layer 222 connected to a fifth activation layer 224, is processed into the reconstructed representation 260.
Regarding Claim 10 Khalid-Miao-Yokouchi
Khalid-Miao-Yokouchi discloses 10. The computer-implemented method of claim 9, wherein the first video portion is generated based on the second video portion that is linked by the second latent representation that is stored in place of the first video portion. (Khalid [0147] ...”. The resulting output (e.g., the embedding 212) is effectively a latent representation (i.e., a compressed frame) of the input data...""[0148] ... recreate the original image ... from the low dimensional embedding." Khalid additionally teaches: "[0165] The decoder portion ... generates the reconstructed representation 260 for each frame 250...")
Miao teaches comparing a current representation to previously stored representations and using the previously stored representation when a match is identified.
Therefore, "wherein the first video portion is generated based on the second video portion that is linked by the second latent representation that is stored in place of the first video portion" is taught by the combination of Khalid and Miao. Khalid teaches generating a reconstructed frame from a stored latent representation ([0147], [0148], [0165]). Miao teaches comparing a current representation to previously stored representations and utilizing the previously stored representation when a match is identified (¶¶ [0037]-[0040]). Therefore, when the current video portion is represented by a previously stored latent representation corresponding to another video portion, reconstruction of the current video portion from that stored latent representation necessarily generates the first video portion based on the second video portion linked to the stored latent representation.
Regarding Claim 19 Khalid-Maio-Yokouchi
Khalid-Mao discloses 19. The computer program product of claim 16,
Khalid-Maio does not explicitly disclose
wherein the first surgical data is deleted after the compressed surgical data is stored.
However, in the same field of endeavor Yokouchi discloses more explicitly the following:
wherein the first surgical data is deleted after the compressed surgical data is stored. (Yokouchi, [0068] “…. The image compressor 136 compresses the input image according to an encoding method such as Joint Photographic Experts Group (JPEG), and thereby generates compressed image data. The image compressor 136 deletes the input image after generating the compressed image data." Accordingly, Yokouchi teaches deleting original data after generating and storing a compressed version thereof.)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Khalid-Maio to delete the original video after the compressed video has been stored as taught by Yokouchi expressly teaches deleting the original image after generating compressed image data in order to reduce storage requirements and avoid retaining redundant uncompressed data ([0068]). Applying this known storage-management technique to Khalid's compressed video representations would have predictably reduced storage consumption while preserving the compressed video data needed for subsequent processing and reconstruction. Such modification would have yielded predictable results.
Claim Rejections - 35 USC § 103
Claims 11-12, 14, and 15 are rejected under 35 U.S.C. §103 as being unpatentable over KHALID in view of Anorga et al. (US-10891485-B2) hereinafter “Anorga” and further in view of Yun et al. (US 11,310,539 B1) hereinafter “Yun” further in further in view of Habibian et al. (US 11,388,416 B2) hereafter “Habibian”.
Regarding Claim 11 Khalid-Anorga-Yun-Habibian
Khalid discloses 11. A system comprising:
a machine learning system comprising one or more machine learning models that are trained to encode a portion of video into a latent representation (Khalid, machine learning architectures for reducing surgical procedure video data set…” [0002], “machine learning architecture includes a dimensionality reduction architecture to discover and extract, from the video data, features which are indicative of surgical instruments, and a sequential relation architecture, to assess a sequence of features which are indicative of surgical instruments and link said features to surgical performance and surgical skill.” [0004][0005][0006] MLA 120A [0314]); and
a data collection system configured to generate a compressed copy of a video catalogue that comprises a plurality of videos, each video in the video catalogue comprising a plurality of video portions (Khalid, “the platform 100A retrieves one or more surgical procedure video data sets indirectly, from database(s) 112.” … “one or more captured and stored surgical procedure video data set” [0115]), wherein generating the compressed copy of the video catalogue comprises:
a) generating a first latent representation of a first video segment of a first video from the video catalogue using the machine learning system; (Khalid, wherein "the embedding 212 is effectively a latent representation (i.e., a compressed frame) of the input data" ([0147]). See also ([0020], [0050]).
However, Khalid does not explicitly disclose:
b) comparing the first latent representation with a plurality of latent representations, wherein each of the plurality of latent representations are corresponding to one or more video segments from one or more videos in the video catalogue;
c) based on a determination that the first latent representation is not similar to any of the plurality of latent representations:
i) storing the first latent representation in the plurality of latent representations; and
ii) linking the first latent representation with the first video segment; and
d) based on a determination that the first latent representation is similar to a second latent representation from the plurality of latent representations,
iii) replacing, in a compressed video corresponding to the first video, the first video segment with the second latent representation.
However, Anorga discloses
b) comparing the first latent representation with a plurality of latent representations, wherein each of the plurality of latent representations are corresponding to one or more video segments from one or more videos in the video catalogue; (Anorga, wherein "similarity of image features can be determined based on whether one or more similarity thresholds are met" ([0048]).)
c) "based on a determination that the first latent representation is not similar to any of the plurality of latent representations:
i) storing the first latent representation in the plurality of latent representations;" (Anorga, wherein previously analyzed image feature vectors are retained and used for subsequent similarity determinations ([0048]).
ii) …
c) "based on a determination that the first latent representation is similar to a second latent representation from the plurality of latent representations, replacing, in a compressed video corresponding to the first video, the first video segment with the second latent representation."(Anorga, wherein duplicate content is identified based on similarity of feature vectors ([0048]. wherein duplicate content is identified based on similarity of feature vectors ([0048]).).
Khalid-Anorga does not explicitly disclose
ii) linking the first latent representation with the first video segment;
iii) replacing, in a compressed video corresponding to the first video, the first video segment with the second latent representation.
However, Yun discloses
ii) linking the first latent representation with the first video segment; (wherein "Embedding data source 138 associates an embedding ... with the video item from which the embedding was generated" and "the key is a video item identifier and the value is the embedding of that video item." (Yun, Col. 5 lines 25-end )
Therefore, A POSITA at the time the application was filed would have been motivated to incorporate the similarity-based duplicate detection techniques of Anorga into Khalid's latent-representation framework in order to reduce storage requirements by avoiding storage of redundant video content. Further, a POSITA would have been motivated to incorporate Yun's embedding association techniques to maintain correspondence between latent representations and their source video segments, thereby facilitating efficient storage, retrieval, and management of compressed video content. Such modification would have yielded predictable results.
Khalid-Anorga-Yun does not explicitly disclose
iii) replacing, in a compressed video corresponding to the first video, the first video segment with the second latent representation.
However, Habibian discloses
iii) replacing, in a compressed video corresponding to the first video, the first video segment with the second latent representation. (Habibian, Col, 10, lines 32-35 “… auto-encoder 401 may be trained using a set of training videos. Encoder 402 in auto-encoder 401 may take a first training video (designated x) and map the first training video to a code z in a latent code space.’’)
Therefore, it would have been obvious to a person having ordinary skill in the art before
the effective filing date of the application to modify the teachings of Khalid-Anorga-Yun with Habibian to create the system of Khalid-Anorga-Yun as outlined above in order to replace in a compressed video corresponding to the first video, the first video segment with the second latent representation.” as suggested by Habibian.
One of ordinary skill in the art would have been motivated to replace a video segment with a latent representation to reduce the amount of data required for storage and transmission while preserving information sufficient to reconstruct the input frames. (Khalid, [0166])
Note: The motivation that was utilized in the rejection of claim 11, applies equally as well to claims 12,14, and 15.
Regarding Claim 12 Khalid-Anorga-Yun-Habibian
Khalid-Anorga-Yun-Habibian discloses 12. The system of claim 11, wherein the plurality of videos in the video catalogue are recordings of different surgical procedures (Khalid [0193]” In one non-limiting example, the first data set consists of 114 full length videos of minimally invasive surgical procedures..." (¶193).) with matching metadata (Yun, generating embeddings from textual tokens associated with video items, including metadata such as titles, descriptions, tags, and transcripts, and matching video items based upon similarity of the resulting embeddings (col. 5, lines 42-67; col. 6, line 61 through col. 7, line 14) ). Accordingly, Khalid teaches recordings of different surgical procedures, while Yun teaches matching metadata associated with the videos.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to organize the plurality of surgical-procedure videos of Khalid using matching metadata as taught by Yun because Yun teaches generating embeddings from metadata associated with video items and using such information to identify similar or related videos. Incorporating metadata-based matching into Khalid's collection of surgical-procedure videos would facilitate efficient retrieval, comparison, indexing, and organization of surgical video content. Such modification would have yielded predictable results.
Regarding Claim 14 Khalid-Anorga-Yun-Habibian
Khalid-Anorga-Yun-Habibian discloses 14. The system of claim 11, wherein linking the first latent representation with the first video segment comprises: storing the first video segment as a video file; and linking the first latent representation with the video file. (Yun, that an embedding is "associated" with the video item from which it was generated and is "stored in association with the corresponding video item" (col. 5, lines 25-35).)
Regarding Claim 15 Khalid-Anorga-Yun-Habibian
Khalid-Anorga-Yun-Habibian discloses 15. The system of claim 14, wherein, based on the determination that the first latent representation is not similar to any of the plurality of latent representations, the first video segment is replaced, in the compressed video corresponding to the first video, with the first latent representation. (Habibian, Col, 10, lines 32-35 “… auto-encoder 401 may be trained using a set of training videos. Encoder 402 in auto-encoder 401 may take a first training video (designated x) and map the first training video to a code z in a latent code space…” Col, 11, lines 3-6 “Code model 404 receives the code z representing an encoded video or portion thereof and generates a probability distribution P(z) over a set of compressed codewords that can be used to represent the code z.”)
Claim Rejections - 35 USC § 103
Claim 13 is rejected under 35 U.S.C. §103 as being unpatentable over Khalid-Anorga-Yun-Habibian in view of Yokouchi et al. (US 20140347704 A1) hereinafter “Yokouchi ”
Regarding Claim 13 Khalid-Anorga-Yun-Habibian- Yokouchi
Khalid-Anorga-Yun-Habibian discloses 13. The system of claim 11,
. . .
reconstructing a version of the first video based on the compressed video in response to a playback request. . (Khalid discloses that a decoder "reconstructs frames from the compressed frames generated by the encoder portion" ([0020]). See also Khalid ([0148], [0165]).
Khalid-Anorga-Yun-Habibian does not explicitly disclose wherein the first video is deleted after the compressed video is stored, and
However, Yokouchi discloses wherein the video is deleted after the compressed video is stored. (Yokouchi, [0068] “The image compressor 136 compresses the input image according to an encoding method such as Joint Photographic Experts Group (JPEG), and thereby generates compressed image data. The image compressor 136 deletes the input image after generating the compressed image data." Accordingly, Yokouchi teaches deleting original data after generating and storing a compressed version thereof.)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Khalid-Anorga-Yun to delete the original video after the compressed video has been stored as taught by Yokouchi expressly teaches deleting the original image after generating compressed image data in order to reduce storage requirements and avoid retaining redundant uncompressed data ([0068]). Applying this known storage-management technique to Khalid's compressed video representations would have predictably reduced storage consumption while preserving the compressed video data needed for subsequent processing and reconstruction. Such modification would have yielded predictable results.
Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Dawn et al. (US-10504001-B1)
Bhorkar et al. (US-20200244969-A1)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ASTEWAYE GETTU ZEWEDE whose telephone number is (703)756-1441. The examiner can normally be reached 8:30 am-5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Vaughn, William can be reached on 571-273-8300. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ASTEWAYE GETTU ZEWEDE/Examiner, Art Unit 2481 /WILLIAM C VAUGHN JR/Supervisory Patent Examiner, Art Unit 2481
Prosecution Insights