DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged that application claims priority to foreign application with application number EP22183987.1 dated 07/08/2022. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Information Disclosure Statement
The IDS dated 1/7/2025, 7/24/2025, 1/27/2026, and 7/13/2026 have been considered and placed in the application file.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 21-26 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-3, 5-7 of copending Application No. 18348785 (reference application). Although the claims at issue are not identical, they are not patentably distinct from each other because claims 21-26 of the instant application are anticipated by claims 1-3 and 5-7, respectively, of copending application 18348785, as follows:
Claim 21/Claim 1
wherein the system is configured to perform operations to perform a plurality of tasks on a set of images, the operations comprising:receiving a plurality of tasks, wherein at least one task of the plurality of tasks is associated with a request to identify at least one feature of interest in the set of images;analyzing, using a local spatio-temporal processing module, a subset of images of the set of images to identify the presence of characteristics associated with the at least one feature of interest; and iterating execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images.
receiving the plurality of tasks, wherein at least one task of the plurality of tasks is associated with a request to identify at least one feature of interest in the set of images; analyzing, using a local spatio-temporal processing module, a subset of images of the set of images to identify presence of characteristics associated with the at least one feature of interest; and iterating execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images.
Claim 22/Claim 2
wherein the local spatio-temporal processing module outputs subsets of analyzed images of the set of images, wherein each subset is associated with a task of the plurality of tasks
wherein the local spatio-temporal processing module outputs subsets of analyzed images of the set of images, wherein each subset is associated with a task of the plurality of tasks
Claim 23/Claim 3
wherein the local spatio- temporal processing module determines the presence of characteristics by determining a vector of quality scores, wherein each quality score in the vector of quality scores corresponds to each image of the subset of the images
wherein the local spatio-temporal processing module determines the presence of characteristics by determining a vector of quality scores, wherein each quality score in the vector of quality scores corresponds to each image of the subset of the images
Claim 24/Claim 5
wherein the local spatio-temporal processing module generates a set of feature vectors for features of interest related to the plurality of tasks
wherein the local spatio-temporal processing module generates a set of feature vectors for features of interest related to the plurality of tasks
Claim 25/Claim 6
wherein the operations further comprise: analyze, using a global spatio-temporal processing module, sets of feature vectors for the subset of images analyzed by the local spatio-temporal processing module
wherein the operations further comprise: analyze, using a global spatio-temporal processing module, sets of feature vectors for the subset of images analyzed by the local spatio-temporal processing module
Claim 26/Claim 7
wherein the operations further comprise: aggregate output of the local spatio-temporal processing module for each task of the plurality of tasks using the timeseries analysis module
wherein the operations further comprise: aggregate output of the local spatio-temporal processing module for each task of the plurality of tasks using the timeseries analysis module
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f), is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f), is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f), because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“determine spatio-temporal information of characteristics related to the at least one feature of interest for subsets of images of video content using a local spatio-temporal processing module” in claims 2 and 21-26;
“determine the spatio-temporal information of all images of the video content using a global spatio-temporal processing module” in claims 2 and 25;
“iterating execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images” in claim 21;
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f), they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f), applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim(s) 1-28 are rejected under 35 U.S.C. 112(b), as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 1 recites “select, using a frame selector, a plurality of images from a group of images in the temporally ordered set of images the group of images bounded by the start image frame and the end image frame, based on an associated score and a quality score of each selected image, wherein the associated score of the selected image indicates a presence of at least one feature of interest.” It is unclear what the Applicant is claiming when stating “quality score”, as quality score can refer to anything (i.e. image quality, how well the image was captured, etc.). For examination purposes, the examiner will interpret this as a relevance value.
Claim 1 also recites “merge a subset of images from the selected plurality of images based on a matching presence of the at least one feature of interest using an objects descriptor, wherein the subset of images is identified based on spatial and temporal coherence using spatio-temporal information”. It is unclear what the Applicant is claiming when stating “objects descriptor”, as objects descriptor can mean anything (i.e. something that describes an object, the object name, etc.). For examination purposes, the examiner will interpret as containing the feature of interest in question.
Claim 1 also recites “and split the temporally ordered set of images using temporal segmentor in temporal intervals which satisfy a temporal coherence of a selected task”. It is unclear what the Applicant is claiming when stating “temporal coherence”, as coherence by definition means having all parts fit together. For examination purposes, the examiner will interpret this limitation as splitting the ordered set of images into groups of images based on whether or not the feature of interest is contained in the images.
Claim 13 recites “wherein the video summary is generated by selecting relevant frames from the captured images and has a variable frame rate video output has a variable frame rate video output”. It is unclear what the Applicant is claiming when stating the video summary “has a variable frame rate video output has a variable frame rate video output”, as videos by definition contain a certain amount of frames per second. For examination purposes, the examiner will interpret this as the video summary containing a video.
Claim 18 recites “wherein the generated dashboard comprises aggregated information from one or more of the event detector, frame selector, object descriptor, and temporal segmentor”. It is unclear what the Applicant is claiming when stating there are aggregated information from the event detector, frame selector, object descriptor, and temporal segmentor, as it is indefinite what information is being aggregated. For examination purposes, the examiner will interpret this limitation as containing information pertaining to the feature of interest in the image.
Claim 21 recites “wherein the system is configured to perform operations to perform a plurality of tasks on a set of images, the operations comprising: receiving a plurality of tasks, wherein at least one task of the plurality of tasks is associated with a request to identify at least one feature of interest in the set of images; analyzing, using a local spatio-temporal processing module, a subset of images of the set of images to identify the presence of characteristics associated with the at least one feature of interest; and iterating execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images.”. There is insufficient antecedent basis for “set of images”, “feature of interests”, “subset of images”. It is also unclear if these terms and the “numerical score” pertain to the terms as claimed in independent claim 1. It is also unclear if “for each task” pertains to the “plurality of tasks”, as they are lacking antecedent basis. For examination purposes, the examiner will respectively interpret these terms as “the set of images”, “the feature of interests”, “the subset of images”, numerical score as “the associated score”, and “the task of the plurality of tasks”.
Claim 26 recites “aggregate output of the local spatio-temporal processing module for each task of the plurality of tasks using the timeseries analysis module”. It is unclear what is the output of the local spatio-temporal processing module and what is being aggregated as claimed by the Applicant. For examination purposes, the examiner will interpret this as an aggregation of image data.
Claims 27 and 28 correspond to claim 1 and thus are rejected for the same reasons of indefiniteness.
Claims 2-26 are rejected for their dependencies on claim 1.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 21 reciting “wherein the system is configured to perform operations to perform a plurality of tasks on a set of images, the operations comprising: receiving a plurality of tasks, wherein at least one task of the plurality of tasks is associated with a request to identify at least one feature of interest in the set of images; analyzing, using a local spatio-temporal processing module, a subset of images of the set of images to identify the presence of characteristics associated with the at least one feature of interest; and iterating execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images.” is rejected under 35 U.S.C. 112(d) as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. The claim is essentially reciting material that is already stated in independent claim 1 without limiting its scope. For example, claim 21 recites the limitations of identifying a feature of interest and associating this task with a score, both of which are already explicitly recited in claim 1. The examiner has noticed that this claim (and the claims that depend on claim 21) are explicitly taken from application 18348785 (as also compared in the double patenting rejection above). If the Applicant wishes to continue with these set of claims, the examiner recommends amending so that they are in a proper dependent format.
Applicant may cancel the claims, amend the claims to place the claims in proper dependent form, rewrite the claims in independent form, or present a sufficient showing that the dependent claims comply with the statutory requirements.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-28 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1 recites:
“access a temporally ordered set of images from the captured images” is a well-understood, routine, and conventional insignificant extra-solution activity of data gathering.
“detect, using an event detector, an occurrence of an event in the temporally ordered set of images, wherein a start time and an end time of the event are identified by a start image frame and an end image frame in the temporally ordered set of images” which can be reasonably interpreted as a human observer mentally determining an event that is happening in a set of images. A person can also mentally determine the start and end of an image.
“select, using a frame selector, a plurality of images from a group of images in the temporally ordered set of images the group of images bounded by the start image frame and the end image frame, based on an associated score and a quality score of each selected image, wherein the associated score of the selected image indicates a presence of at least one feature of interest” which can be reasonably interpreted as a human observer mentally selecting frames based on how likely they contain a feature of interest.
“merge a subset of images from the selected plurality of images based on a matching presence of the at least one feature of interest using an objects descriptor, wherein the subset of images is identified based on spatial and temporal coherence using spatio-temporal information” which can be reasonably interpreted as a human observer mentally grouping frames based on whether or not they contain a similar feature of interest.
“and split the temporally ordered set of images using temporal segmentor in temporal intervals which satisfy a temporal coherence of a selected task” which can be reasonably interpreted as a human observer mentally splitting frames based on whether or not they contain a similar feature of interest.
Claim 2 recites “determine spatio-temporal information of characteristics related to the at least one feature of interest for subsets of images of video content using a local spatio-temporal processing module and determine the spatio-temporal information of all images of the video content using a global spatio-temporal processing module”. A person can mentally determine characteristics related to a feature of interest and determine spatio-temporal information for an image.
Claim 3 recites “wherein to split the temporally ordered set of images in temporal intervals the one or more processors are further configured to: identify a subset of the temporally ordered set of images with the presence of the at least one feature of interest or identify a subset of the temporally ordered set of images with the presence of an event”. A person can mentally identify a subset of images that contain a feature of interest.
Claim 4 recites “wherein to identify a subset of the temporally ordered set of images with the presence of the at least one feature of interest the one or more processors are further configured to: add bookmarks to images in the temporally ordered set of images, wherein the bookmarked images are part of the subset of the temporally ordered set of images”. A person with the aid of pen and paper can mentally make note of certain images (similar to a bookmark function).
Claim 5 recites “wherein to identify a subset of the temporally ordered set of images with the presence of the at least one feature of interest the one or more processors are further configured to: extract a set of images from the subset of the temporally ordered set of images”. A person can mentally extract a set of images from a subset of images.
Claim 6 recites “wherein the extracted set of images includes characteristics related to the at least one feature of interest”. A person can mentally determine characteristics related to a feature of interest.
Claim 7 recites “wherein to identify a subset of the temporally ordered set of images with the presence of the at least one feature of interest the one or more processors are further configures to: add color to a portion of a timeline of the captured images that matches the subset of the temporally ordered set of images”. A person with the aid of pen and paper can mentally color in a timeline of a video that pertains to a set of images.
Claim 8 recites “wherein the color varies with a level of relevance of an image of the subset of the temporally ordered set of images for the at least one feature of interest”. A person with the aid of pen and paper can mentally provide different shadings/colorings for images that contain feature of interest.
Claim 9 recites “wherein the color varies with a level of relevance of an image of the subset of the temporally ordered set of images for characteristics related to the at least one feature of interest”. A person with the aid of pen and paper can mentally provide different shadings/colorings for images that contain characteristics of a feature of interest.
Claim 10 recites “wherein the color differs with different features of interest related to the at least one feature of interest”. A person with the aid of pen and paper can mentally provide different shadings/colorings for images that contain feature of interest.
Claim 11 recites “wherein the color differs with different events detected using the event detector”. A person with the aid of pen and paper can mentally provide different shadings/colorings for images with different events.
Claim 12 recites “wherein the timeline is presented as part of a video summary, wherein the video summary includes overlaid text and graphics”. A person with the aid of pen and paper can mentally provide a summary with drawings.
Claim 13 recites “wherein the video summary is generated by selecting relevant frames from the captured images and has a variable frame rate video output has a variable frame rate video output”. This is a well-understood, routine, and conventional insignificant extra-solution activity of data gathering.
Claim 14 recites “wherein the occurrence of the event represents a portion of a medical procedure”. This is generally linking the use of the judicial exception to the medical field of use.
Claim 15 recites “wherein the operations further comprise: generate a dashboard with summary of the temporally ordered set of images wherein the summary includes images selected using the frame selector and augmented with display markings”. A person with the aid of pen and paper can create a dashboard with markings for a set of images.
Claim 16 recites “wherein the generated dashboard includes quality scores of a medical procedure performed while images are captured using the imaging device”. A person with the aid of pen and paper can create a dashboard with scores on how well a medical procedure is going. Images captured by an imaging device are a well-understood, routine, and conventional insignificant extra-solution activity of data gathering.
Claim 17 recites “wherein the generated dashboard includes quality scores of an operator of the imaging device performing a medical procedure”. A person with the aid of pen and paper can create a dashboard with scores on how well an operator is performing a medical procedure.
Claim 18 recites “wherein the generated dashboard comprises aggregated information from one or more of the event detector, frame selector, object descriptor, and temporal segmentor”. A person with the aid of pen and paper can create a dashboard with aggregated information pertaining to the image.
Claim 19 recites “wherein the temporally ordered set of images are received directly from the imaging device during a medical procedure”. This is a well-understood, routine, and conventional insignificant extra-solution activity of data outputting.
Claim 20 recites “wherein a presence of at least one feature of interest is determined from a portion of the captured images”. A person can mentally determine if a feature of interest is in a captured image.
Claim 21 recites “wherein the system is configured to perform operations to perform a plurality of tasks on a set of images, the operations comprising: receiving a plurality of tasks, wherein at least one task of the plurality of tasks is associated with a request to identify at least one feature of interest in the set of images; analyzing, using a local spatio-temporal processing module, a subset of images of the set of images to identify the presence of characteristics associated with the at least one feature of interest; and iterating execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images”. A person can receive a task and mentally identify a feature of interest in a set of images and identify characteristics within the feature of interest. A person can also mentally assign a score to each task and rate them on how difficult they are.
Claim 22 recites “wherein the local spatio-temporal processing module outputs subsets of analyzed images of the set of images, wherein each subset is associated with a task of the plurality of tasks”. A person can mentally assign a task for each set of images.
Claim 23 recites “wherein the local spatio- temporal processing module determines the presence of characteristics by determining a vector of quality scores, wherein each quality score in the vector of quality scores corresponds to each image of the subset of the images”. This is a well-understood routine, and conventional insignificant extra-solution activity of feature vectors in machine learning.
Claim 24 recites “wherein the local spatio-temporal processing module generates a set of feature vectors for features of interest related to the plurality of tasks”. This is a well-understood routine, and conventional insignificant extra-solution activity of feature vectors in machine learning.
Claim 25 recites “wherein the operations further comprise: analyze, using a global spatio-temporal processing module, sets of feature vectors for the subset of images analyzed by the local spatio-temporal processing module”. A person can analyze a set of feature vectors of an image.
Claim 26 recites “wherein the operations further comprise: aggregate output of the local spatio-temporal processing module for each task of the plurality of tasks using the timeseries analysis module”. A person with the aid of pen and paper can aggregate information pertaining to a task.
Claim 27 corresponds to claim 1, and thus is rejected for the same abstract reasons as claim 1.
Claim 28 correspond to claim 1, additionally reciting a non-transitory computer readable medium respectively. These additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limitations on practicing the abstract idea. The remaining limitations are rejected for the same abstract reasons as claim 1.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-6, 14, 19-22, 26-28 are rejected under 35 U.S.C. 102(a)(1) and (a)(2) as being anticipated by Park (US 20110301447 A1).
Regarding claim 1, Park discloses a computer-implemented system (Park, paragraph [0083], "FIG. 1 depicts a schematic of the video interpretation system of the current invention.") comprising:
one or more memory devices storing processor-executable instructions and one or more processors configured to execute the processor-executable instructions (Park, paragraph [0236], "Actions include, but are not limited to, video storage on a computer medium (such as a hard disk, thumb drive, picture archiving and communication system (PACS), or otherwise) and video playback for review by the physician."), to cause the system to perform operations for spatio-temporal analysis of images captured with an imaging device the operations comprising:
access a temporally ordered set of images from the captured images (Park, paragraph [0219], "This system will automatically interpret any video data, output annotations and classifications according to the minimal standard terminology for endoscopy and other features, including but not limited to, frame degradation factors such as obstructions, blur, glare, and illumination, and objects in the colonoscopic video scene such as blood, stool, water, and tools"),
detect, using an event detector, an occurrence of an event in the temporally ordered set of images (Park, paragraph [0212], " The detection and tracking of polyps or diverticula are displayed in FIG. 11")
wherein a start time and an end time of the event are identified by a start image frame and an end image frame in the temporally ordered set of images (Park, paragraph [0227], "Continue to apply step 2 to subsequent video frames until the end of the video is reached", a video has a start and end frame),
select, using a frame selector, a plurality of images from a group of images in the temporally ordered set of images the group of images bounded by the start image frame and the end image frame (Park, paragraph [0224], “Once each individual frame is scored for each particular type of content under analysis, it is necessary to aggregate the video frames into discrete video sections of similar content measure”), based on an associated score and a quality score of each selected image, wherein the associated score of the selected image indicates a presence of at least one feature of interest (Park, paragraph [0221], "For any embodiment, the content measure for a particular feature reflects how much of the feature is present in the analyzed frames. This content measure can be a simple binary score of either "true" or "false". Alternatively, the content score may incorporate the uncertainties inherent in any measurement by producing a probability value (0% to 100%) describing to what extent one or more features may be visible in the frame. In addition, the content score can take into account the clinical relevance of the feature, assigning a relevance value (0% to 100%) as to whether the features are important to the physician."),
merge a subset of images from the selected plurality of images based on a matching presence of the at least one feature of interest using an objects descriptor, wherein the subset of images is identified based on spatial and temporal coherence using spatio-temporal information (Park, paragraph [0226], "Check the subsequent frame analysis result against the video section category. If they are the same, consider the frame to be part of the current section. If they are different, create a new video section starting with the new frame by marking the new frame as the end of the current section and the start of a new section, and categorize the new section according to the new frame's analysis result."),
and split the temporally ordered set of images using temporal segmentor in temporal intervals which satisfy a temporal coherence of a selected task (Park, paragraph [0229], "FIG. 13 graphically depicts this embodiment for N=2 with X illustrating video frames "containing content type X" and O illustrating vide frames "not containing content type X".", aggregation splits the images).
Regarding claim 2, Park discloses the system of claim 1, wherein the one or more processors are further configured to: determine spatio-temporal information of characteristics related to the at least one feature of interest for subsets of images of video content using a local spatio-temporal processing module (Park, paragraph [0212], "The detection of the polyp or diverticulum can be indicated with a black ellipse as shown in FIG. 11(b)). The tracking phase can be indicated with increasing line thicknesses of the ellipse as illustrated in FIG. 11(c) and FIG. 11(d). Of course, other shapes, size and color schemes for alerts can also be used."),
and determine the spatio-temporal information of all images of the video content using a global spatio-temporal processing module (Park, paragraph [0198], "Once a polyp or diverticulum is detected in a video frame, tracking is enabled to track the polyp or diverticulum in subsequent video frames").
Regarding claim 3, Park discloses the system of claim 1, wherein to split the temporally ordered set of images in temporal intervals the one or more processors are further configured to: identify a subset of the temporally ordered set of images with the presence of the at least one feature of interest or identify a subset of the temporally ordered set of images with the presence of an event (Park, paragraph [0229], "FIG. 13 graphically depicts this embodiment for N=2 with X illustrating video frames "containing content type X" and O illustrating vide frames "not containing content type X".").
Regarding claim 4, Park discloses the system of claim 3, wherein to identify a subset of the temporally ordered set of images with the presence of the at least one feature of interest the one or more processors are further configured to: add bookmarks to images in the temporally ordered set of images, wherein the bookmarked images are part of the subset of the temporally ordered set of images (Park, paragraph [0248], "Step 1 serves to "tag" a number of frames with absolute spatial location information. Though it is not a restriction of this process, it is often considered that these tags are quite accurate, but coarsely spaced both spatially and temporally. Anatomical landmarks during colonoscopy are an excellent example of this process step: as the live video is displayed during capture, relevant "landmarks" within the colon, such as the anus (20), splenic flexure (22), hepatic flexure (23), ileocecal valve (24), and/or appendiceal orifice (25) as illustrated in FIG. 6 can be marked").
Regarding claim 5, Park discloses the system of claim 3, wherein to identify a subset of the temporally ordered set of images with the presence of the at least one feature of interest the one or more processors are further configured to: extract a set of images from the subset of the temporally ordered set of images (Park, paragraph [0236], "Actions include, but are not limited to, video storage on a computer medium (such as a hard disk, thumb drive, picture archiving and communication system (PACS), or otherwise) and video playback for review by the physician.", video playback is a form of extracting sets of images).
Regarding claim 6, Park discloses the system of claim 5, wherein the extracted set of images includes characteristics related to the at least one feature of interest (Park, paragraph [0202], "Region refinement preferably starts by calculating region properties based on their area, average intensity, average color value, solidity, and eccentricity. Regions that satisfy a list of pre-modeled criteria proceed with a shape and texture identification").
Regarding claim 14, Park discloses the system of claim 1, wherein the occurrence of the event represents a portion of a medical procedure (Park, paragraph [0212], " The detection and tracking of polyps or diverticula are displayed in FIG. 11").
Regarding claim 19, Park discloses the system of claim 1, wherein the temporally ordered set of images are received directly from the imaging device during a medical procedure (Park, paragraph [0192], "The concept of the digital colon model can be augmented by, and in addition to, video data acquired using different macroscopic imaging modalities, including data from microscopic and spectroscopic probe systems, such as confocal microscopy, optical coherence tomography, and infrared spectroscopy").
Regarding claim 20, Park discloses the system of claim 1, wherein a presence of at least one feature of interest is determined from a portion of the captured images (Park, paragraph [0212], " The detection and tracking of polyps or diverticula are displayed in FIG. 11").
Regarding claim 21, Park discloses the system of claim 1,wherein the system is configured to perform operations to perform a plurality of tasks on a set of images, the operations comprising: receiving a plurality of tasks, wherein at least one task of the plurality of tasks is associated with a request to identify at least one feature of interest in the set of images (Park, paragraph [0212], " The detection and tracking of polyps or diverticula are displayed in FIG. 11"),
analyzing, using a local spatio-temporal processing module, a subset of images of the set of images to identify the presence of characteristics associated with the at least one feature of interest (Park, paragraph [0202], "Region refinement preferably starts by calculating region properties based on their area, average intensity, average color value, solidity, and eccentricity. Regions that satisfy a list of pre-modeled criteria proceed with a shape and texture identification."),
and iterating execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images (Park, paragraph [0221], "For any embodiment, the content measure for a particular feature reflects how much of the feature is present in the analyzed frames. This content measure can be a simple binary score of either "true" or "false". Alternatively, the content score may incorporate the uncertainties inherent in any measurement by producing a probability value (0% to 100%) describing to what extent one or more features may be visible in the frame. In addition, the content score can take into account the clinical relevance of the feature, assigning a relevance value (0% to 100%) as to whether the features are important to the physician.").
Regarding claim 22, Park discloses the system of claim 21, wherein the local spatio-temporal processing module outputs subsets of analyzed images of the set of images, wherein each subset is associated with a task of the plurality of tasks (Park, paragraph [0229], "FIG. 13 graphically depicts this embodiment for N=2 with X illustrating video frames "containing content type X" and O illustrating vide frames "not containing content type X".").
Regarding claim 26, Park discloses the system of claim 21, wherein the operations further comprise: aggregate output of the local spatio-temporal processing module for each task of the plurality of tasks using the timeseries analysis module (Park, paragraph [0229], "FIG. 13 graphically depicts this embodiment for N=2 with X illustrating video frames "containing content type X" and O illustrating vide frames "not containing content type X".").
Claim 27 corresponds to claim 1. Thus, they are rejected for the same reasons of anticipation as claim 1.
Claim 28 corresponds to claim 1, additionally reciting a non-transitory computer readable medium including instructions (Park, paragraph [0236], “Actions include, but are not limited to, video storage on a computer medium (such as a hard disk, thumb drive, picture archiving and communication system (PACS), or otherwise) and video playback for review by the physician.”). Thus, they are rejected for the same reasons of anticipation as claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 7-12 are rejected under 35 U.S.C. 103 as being unpatentable over Park (US 20110301447 A1) in view of Linard (US 20180286049 A1).
Regarding claim 7, Park discloses the system of claim 3.
Park does not teach “wherein to identify a subset of the temporally ordered set of images with the presence of the at least one feature of interest the one or more processors are further configures to: add color to a portion of a timeline of the captured images that matches the subset of the temporally ordered set of images”.
However, Linard teaches wherein to identify a subset of the temporally ordered set of images with the presence of the at least one feature of interest the one or more processors are further configures to: add color to a portion of a timeline of the captured images that matches the subset of the temporally ordered set of images (Linard, paragraph [0070], Fig. 2 below, "FIG. 2 illustrates the case of an output discretized into four distinct values, each of them being represented by a distinct color: white 24, light gray (dot) 28, dark gray (wave) 22 or black (hatched) 26. If there is an order relation between the output values, this order can be kept between gray levels to which the values are mapped. If not a random order may be chosen. These four gray values may indicate for example four interpretability levels ordered as: not interpretable at all, hardly interpretable, partially interpretable, fully interpretable. They may also indicate for example: not sufficiently interpretable, sufficiently interpretable and belonging to tissue type A, sufficiently interpretable and belonging to tissue type B, sufficiently interpretable and belonging to tissue type C, where there is no order relation between these three tissue types.”).
PNG
media_image1.png
336
316
media_image1.png
Greyscale
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to generate a timeline of Park’s images and color code them based on their detected feature and whether they were detected, as taught by Linard.
The suggestion/motivation for doing so would have been to allow medical users to easily jump back and forth for real-time video analysis.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Park in view of Linard to obtain the invention as specified in claim 7.
Regarding claim 8, Park in view of Linard discloses the system of claim 7, wherein the color varies with a level of relevance of an image of the subset of the temporally ordered set of images for the at least one feature of interest (Linard, paragraph [0070], Fig. 2 below, "FIG. 2 illustrates the case of an output discretized into four distinct values, each of them being represented by a distinct color: white 24, light gray (dot) 28, dark gray (wave) 22 or black (hatched) 26. If there is an order relation between the output values, this order can be kept between gray levels to which the values are mapped. If not a random order may be chosen. These four gray values may indicate for example four interpretability levels ordered as: not interpretable at all, hardly interpretable, partially interpretable, fully interpretable. They may also indicate for example: not sufficiently interpretable, sufficiently interpretable and belonging to tissue type A, sufficiently interpretable and belonging to tissue type B, sufficiently interpretable and belonging to tissue type C, where there is no order relation between these three tissue types.", as modified above, the coloring varies based on whether the features were detected).
Regarding claim 9, Park in view of Linard discloses the system of claim 7, wherein the color varies with a level of relevance of an image of the subset of the temporally ordered set of images for characteristics related to the at least one feature of interest (Linard, paragraph [0070], Fig. 2 below, "FIG. 2 illustrates the case of an output discretized into four distinct values, each of them being represented by a distinct color: white 24, light gray (dot) 28, dark gray (wave) 22 or black (hatched) 26. If there is an order relation between the output values, this order can be kept between gray levels to which the values are mapped. If not a random order may be chosen. These four gray values may indicate for example four interpretability levels ordered as: not interpretable at all, hardly interpretable, partially interpretable, fully interpretable. They may also indicate for example: not sufficiently interpretable, sufficiently interpretable and belonging to tissue type A, sufficiently interpretable and belonging to tissue type B, sufficiently interpretable and belonging to tissue type C, where there is no order relation between these three tissue types.", as modified above, the coloring varies based on what features were detected. Different features will have different characteristics).
Regarding claim 10, Park in view of Linard discloses the system of claim 7, wherein the color differs with different features of interest related to the at least one feature of interest (Linard, paragraph [0070], Fig. 2 below, "FIG. 2 illustrates the case of an output discretized into four distinct values, each of them being represented by a distinct color: white 24, light gray (dot) 28, dark gray (wave) 22 or black (hatched) 26. If there is an order relation between the output values, this order can be kept between gray levels to which the values are mapped. If not a random order may be chosen. These four gray values may indicate for example four interpretability levels ordered as: not interpretable at all, hardly interpretable, partially interpretable, fully interpretable. They may also indicate for example: not sufficiently interpretable, sufficiently interpretable and belonging to tissue type A, sufficiently interpretable and belonging to tissue type B, sufficiently interpretable and belonging to tissue type C, where there is no order relation between these three tissue types.", as modified above, the coloring varies based on what features were detected).
Regarding claim 11, Park in view of Linard discloses the system of claim 7, wherein the color differs with different events detected using the event detector (Linard, paragraph [0070], Fig. 2 below, "FIG. 2 illustrates the case of an output discretized into four distinct values, each of them being represented by a distinct color: white 24, light gray (dot) 28, dark gray (wave) 22 or black (hatched) 26. If there is an order relation between the output values, this order can be kept between gray levels to which the values are mapped. If not a random order may be chosen. These four gray values may indicate for example four interpretability levels ordered as: not interpretable at all, hardly interpretable, partially interpretable, fully interpretable. They may also indicate for example: not sufficiently interpretable, sufficiently interpretable and belonging to tissue type A, sufficiently interpretable and belonging to tissue type B, sufficiently interpretable and belonging to tissue type C, where there is no order relation between these three tissue types.", as modified above, different features detected is a different event).
Regarding claim 12, Park in view of Linard discloses the system of claim 7.
Park in view of Linard does not teach “wherein the timeline is presented as part of a video summary, wherein the video summary includes overlaid text and graphics”.
However, Linard additionally teaches wherein the timeline is presented as part of a video summary, wherein the video summary includes overlaid text and graphics (Linard, paragraph [0087], Fig. 4B below, " FIG. 4A and FIG. 4B illustrate the case where the second algorithm is a content-based video retrieval processing that has been launched on the current temporal region of the video of interest. The output created by this second algorithm and displayed to the user consists of three reference videos (41, 42, 43) together with their annotations (44, 45, 46), where the annotations include for example the diagnostic class of the reference video").
PNG
media_image2.png
347
449
media_image2.png
Greyscale
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to include the video frames and annotate Park’s (in view of Linard) timeline with their detection data, as additionally taught by Linard.
The suggestion/motivation for doing so would have been to create a compact view for medical users to quickly grasp the situation in a medical procedure.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Park in view of Linard with the additional teachings of Linard to obtain the invention as specified in claim 12.
Regarding claim 13, Park in view of Linard discloses the system of claim 12, wherein the
video summary is generated by selecting relevant frames from the captured images and has a variable frame rate video output has a variable frame rate video output (Linard, paragraph [0088], " FIG. 4A and FIG. 4B illustrate the case where the second algorithm is a content-based video retrieval processing that has been launched on the current temporal region of the video of interest. The output created by this second algorithm and displayed to the user consists of three reference videos (41, 42, 43) together with their annotations (44, 45, 46), where the annotations include for example the diagnostic class of the reference video. These reference videos have been extracted from an external database as the most visually similar to the set of contiguous images associated with the current temporal region selected by the cursor 15 in FIG. 4A", videos naturally have a frame output).
Claim(s) 15-18 are rejected under 35 U.S.C. 103 as being unpatentable over Park (US 20110301447 A1) in view of Kumar (US 20120316421 A1).
Regarding claim 15, Park discloses the system of claim 1.
Park does not teach “wherein the operations further comprise: generate a dashboard with summary of the temporally ordered set of images wherein the summary includes images selected using the frame selector and augmented with display markings”.
However, Kumar teaches wherein the operations further comprise: generate a dashboard with summary of the temporally ordered set of images wherein the summary includes images selected using the frame selector and augmented with display markings (Kumar, paragraph [0208], Fig. 18 below, "FIG. 18 depicts an illustrative screen shot of a user interface application 1800 designed to support review of imaging data").
PNG
media_image3.png
394
494
media_image3.png
Greyscale
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to generate an interface that allows a user to view Park’s images and the data (such as detected features and scores) in Park’s images, as taught by Kumar.
The suggestion/motivation for doing so would have been to provide a compact view for medical users to quickly grasp the situation in a medical procedure.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Park in view of Kumar to obtain the invention as specified in claim 15.
Regarding claim 16, Park in view of Kumar discloses the system of claim 15, wherein the generated dashboard includes quality scores of a medical procedure performed while images are captured using the imaging device (Park, paragraph [0221], "For any embodiment, the content measure for a particular feature reflects how much of the feature is present in the analyzed frames. This content measure can be a simple binary score of either "true" or "false". Alternatively, the content score may incorporate the uncertainties inherent in any measurement by producing a probability value (0% to 100%) describing to what extent one or more features may be visible in the frame. In addition, the content score can take into account the clinical relevance of the feature, assigning a relevance value (0% to 100%) as to whether the features are important to the physician.").
Regarding claim 17, Park in view of Kumar discloses the system of claim 15, wherein the generated dashboard includes quality scores of an operator of the imaging device performing a medical procedure (Park, paragraph [0221], "For any embodiment, the content measure for a particular feature reflects how much of the feature is present in the analyzed frames. This content measure can be a simple binary score of either "true" or "false". Alternatively, the content score may incorporate the uncertainties inherent in any measurement by producing a probability value (0% to 100%) describing to what extent one or more features may be visible in the frame. In addition, the content score can take into account the clinical relevance of the feature, assigning a relevance value (0% to 100%) as to whether the features are important to the physician.").
Regarding claim 18, Park in view of Kumar discloses the system of claim 15, wherein the generated dashboard comprises aggregated information from one or more of the event detector, frame selector, object descriptor, and temporal segmentor (Kumar, paragraph [0208], Fig. 18 below, "FIG. 18 depicts an illustrative screen shot of a user interface application 1800 designed to support review of imaging data", as modified above, the interface contains data).
Claim(s) 23-25 are rejected under 35 U.S.C. 103 as being unpatentable over Park (US 20110301447 A1) in view of Bhatia (US 20150379373 A1).
Regarding claim 23, Park discloses the system of claim 21.
Park does not teach “wherein the local spatio-temporal processing module determines the presence of characteristics by determining a vector of quality scores, wherein each quality score in the vector of quality scores corresponds to each image of the subset of the images”.
However, Bhatia teaches wherein the local spatio-temporal processing module determines the presence of characteristics by determining a vector of quality scores, wherein each quality score in the vector of quality scores corresponds to each image of the subset of the images (Bhatia, paragraph [0011], In computing the set of quality scores of each of the image sets, the method includes generating feature vectors corresponding to the features obtained from each of the plurality of image sets")).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to generate a feature vector with a quality score for Park’s images, as taught by Bhatia.
The suggestion/motivation for doing so would have been to provide more detailed training data on determining features of interest.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Park in view of Bhatia to obtain the invention as specified in claim 23.
Regarding claim 24, Park discloses the system of claim 21.
Park does not teach “wherein the local spatio-temporal processing module generates a set of feature vectors for features of interest related to the plurality of tasks”.
However, Bhatia teaches wherein the local spatio-temporal processing module generates a set of feature vectors for features of interest related to the plurality of tasks (Bhatia, paragraph [0011], In computing the set of quality scores of each of the image sets, the method includes generating feature vectors corresponding to the features obtained from each of the plurality of image sets")).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to generate feature vectors for Park’s features, as taught by Bhatia.
The suggestion/motivation for doing so would have been to provide more detailed training data on determining features of interest.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Park in view of Bhatia to obtain the invention as specified in claim 24.
Regarding claim 25, Park discloses the system of claim 21.
Park does not teach “wherein the operations further comprise: analyze, using a global spatio-temporal processing module, sets of feature vectors for the subset of images analyzed by the local spatio-temporal processing module”.
However, Bhatia teaches wherein the operations further comprise: analyze, using a global spatio-temporal processing module, sets of feature vectors for the subset of images analyzed by the local spatio-temporal processing module (Bhatia, paragraph [0011], In computing the set of quality scores of each of the image sets, the method includes generating feature vectors corresponding to the features obtained from each of the plurality of image sets")).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to generate feature vectors for Park’s images, as taught by Bhatia.
The suggestion/motivation for doing so would have been to provide more detailed training data on determining features of interest.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Park in view of Bhatia to obtain the invention as specified in claim 25.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WAYNE ZHANG whose telephone number is (571) 272-0245. The examiner can normally be reached Monday-Friday 10:00-6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ms. Sumati Lefkowitz can be reached on (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WAYNE ZHANG/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
