Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 14 and 30 are objected to under 37 CFR 1.75 as being a substantial duplicate of claims 12 and 28. When two claims in an application are duplicates or else are so close in content that they both cover the same thing, despite a slight difference in wording, it is proper after allowing one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP § 608.01(m).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 7-10, 13, 17-19, 23-26 and 29 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Moutinho (US 2017/0060867 A1).
As to claim 1, Moutinho teaches a method of automatically identifying a feature from a video (A method of finding a match in a match database with a target media; Abstract), the method comprising: receiving, by a processor (Processor; 704 Fig. 7), a reference to be searched (Receive a target media item; 652 Fig. 6B); identifying one or more descriptors from the reference (Extract multiple visual descriptors from the target media item; 654 Fig. 6B); searching for a correlation between the one or more descriptors from the reference and clusters of one or more descriptors from a bulk video frame set (comparing the cluster keys of the target media item with keys of sequences stored in the matching database; Abstract), wherein the bulk video frame set (matching database; 108 Fig. 1) comprises a plurality of sampled video frames from the video (A video is composed by media segments which correspond to a group of frames from the video; [0027]), wherein the clusters of one or more descriptors from the bulk video frame set have been clustered by the one or more descriptors from the bulk video frame set (cluster keys within cluster table; [0027]); selecting one or more images from the bulk video frame set based on the correlation; and outputting the one or more images (Generate a match list; Abstract, 662 Fig. 6B).
Moutinho does not specify the video and image matching is that of a surgical procedure and that the sampled video frames of the bulk video frame set are of the surgical procedure. However, the process used to implement such a matching process could be applied to any video content. It is well known within the art to record video during a surgical procedure for multiple reasons, one of which would be to provide decision support to a surgeon as taught by Wolf et al. (US 2020/0273575 A1).
As to claim 2, Moutinho teaches the method of claim 1, wherein receiving the reference comprises receiving a reference image (a target media includes receiving a target media item, wherein the target media item comprises one or more pictures or videos; Abstract).
As to claim 3, Moutinho teaches the method of claim 1, wherein receiving the reference comprises receiving a reference image of one or more of: an MRI scan image, an x-ray image, a video frame, a photograph, or a combination of any of these (a target media includes receiving a target media item, wherein the target media item comprises one or more pictures or videos; Abstract).
As to claim 7, Moutinho teaches the method of claim 1, further comprising clustering the one or more descriptors from the bulk video frame set (creating cluster keys from the projected descriptors, and generating a list of a number of matches by comparing the cluster keys of the target media item with keys of sequences stored in the matching database; Abstract; note: this demonstrates the cluster keys exists within the database which is interpreted to be the claimed bulk video frame set).
As to claim 8, Moutinho teaches the method of claim 1, wherein outputting the one or more images further comprises modifying the video of a surgical procedure to indicate the reference (the reduction in dimension of the target media item may be performed on the user device, and only the reduced data, e.g., cluster keys, may be transferred or uploaded, to a network server, where a search with a media database may be performed; [0076]).
As to claim 9, Moutinho teaches the method of claim 1, wherein outputting further comprises displaying the one or more images (The server 700 may optionally include a display module 710 that may provide information to a display user interface; [0079]).
As to claim 10, Moutinho teaches the method of claim 1, wherein the clusters of one or more descriptors are hierarchical ((cluster keys within cluster table; [0027]) a table is a way of arranging or ordering therefore it is interpreted to be hierarchical).
As to claim 13, Moutinho teaches the method of claim 1, wherein the bulk video frame set comprises sampled video frames from a portion of the video of the surgical procedure (large databases containing equal or similar images or videos; [0018]).
As to claims 17-19, 23-26 and 29, they are the system claim of claims 1-3, 1-10 and 13 therefore are addressed similarly as above. Also see system of Moutinho (Fig. 7) and Liu (Fig. 2).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4-5, 12, 14-16, 20-21, 28, and 30-10 are rejected under 35 U.S.C. 103 as being unpatentable over Moutinho in view of Liu et al. (US 20160259888 A2 hereinafter Liu).
As to claim 4, Moutinho teaches a strong active-learning component ([0023]) however does not explicitly teach the video and image search method comprises searching using a machine-learning agent. Liu teaches a method for content management of video images of anatomical regions (Title) wherein machine-learning is used to identify non-tissue regions ([0036]) therefore reading on the claimed use of machine-learning agent for searching correlation. It would have been obvious for one ordinary skilled in the art before the time of filing to have combined Moutinho’s video and image match searching process with the machine learning process in order to identify certain types of specific regions within a searched image (Liu [0036]).
As to claim 5, Moutinho does not explicitly teach forming the bulk video frame set by sampling the video frames from the video of the surgical procedure. Liu teaches a method for content management of video images of anatomical regions (Title) wherein surgical imagery is recorded during surgical procedure ([0004]). It would have been obvious for one ordinary skilled in the art before the time of filing to have combined Moutinho’s video and image match searching process with the surgical procedure of Liu as an example of possible usage of video searching.
As to claim 12, Moutinho teaches a video and image match searching process (Title) with a strong active-learning component ([0023]) but does not explicitly teach wherein searching for the correlation comprises performing semantic searching. Liu teaches a method for content management of video images of anatomical regions (Title) wherein the search uses natural language parser (216 Fig. 2). ]). It would have been obvious for one ordinary skilled in the art before the time of filing to have combined Moutinho’s video and image match searching process with the enhanced ability of natural language parser in order to have better machine-human interaction and resulting in a faster and more reliable search result.
As to claim 14, Moutinho and Liu teach the method of claim 1, wherein the searching for the correlation comprises performing a semantic search (see mapping for claim 12).
As to claim 15, Moutinho teaches a video and image match searching process but does not explicitly teach identifying a surgical stage from the video of the surgical procedure. Liu teaches a method for content management of video images of anatomical regions (Title) further including a surgical scene analyzer which includes a surgical scene analyzer which contains one or more image-processing operations to analyze the video images captured by the image-capturing device. Therefore, Liu reads on the claimed “identifying surgical stage”. It would have been obvious for one ordinary skilled in the art before the time of filing to have combined Moutinho’s video and image match searching process with the scene analyzer in order to better detect different portions of the input video.
As to claims 20-21, 28, and 30-31, they are the system claim of claims 4-5, 12, and 14-15 and therefore are addressed similarly as above. Also see system of Moutinho (Fig. 7) and Liu (Fig. 2).
As to claim 16, Moutinho teaches a method of automatically identifying a feature from a video (A method of finding a match in a match database with a target media; Abstract), the method comprising: receiving, by a processor (Processor; 704 Fig. 7), a reference image to be searched (Receive a target media item; 652 Fig. 6B); identifying one or more descriptors from the reference image (Extract multiple visual descriptors from the target media item; 654 Fig. 6B); searching for a correlation between the one or more descriptors from the reference image and clusters of one or more descriptors from a bulk video frame set (comparing the cluster keys of the target media item with keys of sequences stored in the matching database; Abstract), wherein the bulk video frame set (matching database; 108 Fig. 1) comprises a plurality of sampled video frames from the video (A video is composed by media segments which correspond to a group of frames from the video; [0027]) that have each been translated into the one or more descriptors and clustered by the one or more descriptors from the bulk video frame set (cluster keys within cluster table; [0027]). Moutinho does not specify the video and image matching is that of a surgical procedure and that the sampled video frames of the bulk video frame set are of the surgical procedure. However, the process used to implement such a matching process could be applied to any video content. It is well known within the art to record video during a surgical procedure for multiple reasons, one of which would be to provide decision support to a surgeon as taught by Wolf et al. (US 2020/0273575 A1).
Moutinho does not explicitly teach further wherein the plurality of sampled video frames have been paired with a set of metadata; selecting one or more images from the bulk video frame set based on the correlation; and outputting the one or more images and their corresponding metadata for display. Liu teaches a method for content management of video images of anatomical regions (Title) where the surgical scene analyzer 210 may generate the metadata associated with the video images after the analysis of the respective video images. The surgical scene analyzer 210 may be further configured to store the metadata in the memory 206 and/or the video database 106 ([0066]) then further the retrieved one or more video portions may be displayed ([0017]). It would have been obvious for one ordinary skilled in the art before the time of filing to have combined Moutinho’s video and image match searching process with Liu’s metadata association method as another way to retrieve video.
Claims 6 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Moutinho in view of Kumar et al. (US 20230245753 A1 hereinafter Kumar)
As to claim 6, Moutinho teaches a video and image match searching process (Title) with a strong active-learning component ([0023]) but does not explicitly teach wherein the plurality of sampled video frames from the video of the surgical procedure forming the bulk video frame set have been sampled at a frame rate of between 1 and 10 frames per second. Kumar teaches a method for AI-assisted surgery (Title) wherein the arthroscope generates consecutive images at a rate of 10 frames per second ([0115]), this reads on the claimed frame rate since the arthroscope is a method of generating the surgical video. It would have been obvious for one ordinary skilled in the art before the time of filing to have combined Moutinho’s video and image match searching process with Kumar’s input video rate because it is a well-known method to input video data.
As to claim 22, it is the system claim of claim 6 and therefore are addressed similarly as above. Also see system of Moutinho (Fig. 7).
Claims 11 and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Moutinho in view of Yao et al. (US 2020/0394499 A1 hereinafter Yao).
As to claim 11, Moutinho teaches a video and image match searching process (Title) with a strong active-learning component ([0023]) but does not explicitly teach wherein identifying one or more descriptors from the reference comprises using inputs to a last layer of a neural network applied to the reference to identify fc7 descriptors. Yao teaches identifying complex events wherein CNNs may be used for and FC7 features and layer are used ([0078]-[0079]). It would have been obvious for one ordinary skilled in the art before the time of filing to have combined Moutinho’s video and image match searching process with an active learning component with Yao’s use of CNN and FC7 in order to enhance the detection and identification of complex events.
As to claim 27, it is the system claim of claim 11 and therefore are addressed similarly as above. Also see system of Moutinho (Fig. 7).
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CLAIRE X WANG whose telephone number is (571)270-1051. The examiner can normally be reached M-F 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Yvonne Eyler can be reached at (571) 272-1200. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
CLAIRE X. WANG
Supervisory Patent Examiner
Art Unit 1774
/CLAIRE X WANG/Supervisory Patent Examiner, Art Unit 1774