Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
In particular, independent claim 1 recites "dynamically modifying perspective views of the plot in an attempt to identify one or more features," but then the very next clause treats identification as an accomplished fact: "in response to identifying a particular feature, sampling the original set of data..." This creates an internal inconsistency a PHOSITA can't resolve: is successful identification a required step of the method, or is the claim satisfied merely by making the attempt, regardless of the outcome? "Attempt to" language is classic aspirational/intended-result phrasing that MPEP 2173.05(b) and the case law on indefinite functional language (e.g., the "aim to" / "trying to" line of reasoning) treat with suspicion, precisely because it doesn't clearly delineate what act the claim actually requires.
Furthermore, the phrase "to identify a data relationship that exists within the original set of data" is broad and vague. It fails to specify what parameters, variables, or data types the relationship connects, leaving the scope ambiguous to a person of skill in the art. More specifically, the claim demands the identification of a "data relationship," but does not state whether this is a correlation, dependency, or structural link between the disparate data types, specific subsets, or temporal attributes. Additionally, the claim says the relationship "contributed to the feature being detectable," yet it fails to map how a geometric grouping in the visual plot translates back to specific underlying data variables in the original set. Furthermore, the phrase "contributed to" imposes a causal/qualitative standard with no objective boundary. A person of ordinary skill in the art cannot determine with reasonable certainty what specific metrics or comparative thresholds define whether a "data relationship" has been successfully identified to trigger the re-training step.
Independent claims 13 and 19 also have similar issues that require clarification.
Other claims are also rejected under 112(b) by virtue of their dependency.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7, 9-15 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Davidson et al. (Pub. No. US 2021/0081822), in view of Pang et al. (Pat. No. US 9,619,691).
Claim 1
Regarding claim 1, Davidson discloses a method comprising accessing an original set of data having a plurality of disparate data types (Davidson, pars. 24-26 disclose that “the plurality of images 120 can be any of a number of ... images” that, together with associated “metadata 202,” such as multispectral or multimodal images captured across “one to twelve channels,” are received by the computer system; the pixel-based image data and the associated metadata constitute disparate data types).
Davidson further discloses generating a dimensionally reduced data set by performing a dimensional reduction operation on the original set of data, wherein the dimensionally reduced data set is plottable in a coordinate system (Davidson, par. 27 discloses that “dimensionality reduction algorithm 104” receives a multi-dimensional dataset of “image-derived features” output by a first, feature-extracting machine learning algorithm — e.g., 2,048 features per image — and reduces it “to three or fewer dimensions” using PCA, UMAP, or t-SNE).
Davidson further discloses plotting the dimensionally reduced data set in the coordinate system, resulting in generation of a visual plot of the dimensionally reduced data set (Davidson, par. 40 discloses generating a “visual representation 400”, also called an “object map,” that is “a two-dimensional plot with icons representing one or more datapoints ... [or] a three-dimensional plot,” in which “the X axis coordinate is based on a first dimension ... the Y axis coordinate is based on a second dimension, [and] the Z axis coordinate is based on a third dimension” of the dimensionally-reduced dataset. See also Fig. 3, steps 308, 310 and 316).
Davidson further discloses that the one or more features include multiple portions of the dimensionally reduced data set, said multiple portions being grouped together to form the one or more features (Davidson, par. 45 discloses that “clustering algorithm 106” groups datapoints in the dimensionally-reduced dataset into a plurality of clusters, each represented in the visual plot by icons “positioned closed together,” “shaded with a same color,” or “encircled by a polygon”).
Davidson further discloses, in response to identifying a particular feature, sampling the original set of data to identify a data relationship that exists within the original set of data, said data relationship being one that contributed to the feature being detectable (Davidson, par. 50 discloses that selection of an icon or a cluster causes the computer system to display the original images 120 associated with that cluster in a second region of the graphical user interface, from which the user determines and applies a “user classification label” — e.g., “1N” for images depicting one cell nucleus — that identifies the shared characteristic among the sampled images responsible for their having clustered together).
Davidson further discloses triggering a re-training of the computer vision algorithm based on the identified data relationship (Davidson discloses that the applied user classification labels are fed, together with the image-derived features, back into dimensionality reduction algorithm 104 and a “second machine learning algorithm 110” in an iterative loop (See Fig. 1) that generates updated, predicted classification labels, and that the resulting labeled “training dataset 130” is thereafter used to train (or, in successive iterations, re-train) a “target machine learning algorithm” — i.e., an image-recognition, or computer vision, algorithm — to classify images, based on the data relationships identified through the clustering and labeling process).
Davidson, however, does not expressly disclose dynamically modifying perspective views of the plot in an attempt to identify one or more features that are visually detectable via a computer vision algorithm, wherein the multiple portions are grouped together to form the one or more features based on the perspective views of the plot being dynamically modified.
In the same field of endeavor — namely, computer-vision-based identification of features formed by a three-dimensional arrangement of data — Pang discloses projecting a three-dimensional point cloud into a plurality of two-dimensional depth images corresponding to different viewpoints of the point cloud (i.e., dynamically modified perspective views), and applying an object detection algorithm — a computer vision algorithm — to those projected views to detect a target object formed by a grouping of points that becomes visually detectable from a given viewpoint, the detected two-dimensional locations thereafter being re-projected into the three-dimensional space to determine the location of the identified feature (See col. 3, ll. 27-60).
Therefore, it would have been obvious to a person having ordinary skill in the art (“PHOSITA”) before the effective filing date of the claimed invention to incorporate Pang’s technique of dynamically generating and computer-vision-analyzing multiple perspective views of three-dimensionally arranged data into Davidson’s visual analysis platform, so as to automate the identification of clusters/features that Davidson’s system otherwise identifies through a static, algorithmically-fixed clustering step coupled with manual user review. A PHOSITA would have been motivated to make this combination because Davidson itself identifies the problem that manual user review of large image sets “can be time consuming and expensive, especially when a user with specialized training is needed,” (Davidson, par. 4) and because Davidson expressly contemplates that its visual representation “is a three-dimensional plot,” such that applying a known, analogous-art technique for automatically detecting features across multiple viewpoints of three-dimensional data would have yielded the predictable result of reducing the manual exploration burden that Davidson identifies as a shortcoming of the prior art, with a reasonable expectation of success.
Claim 2
Regarding claim 2, the Davidson/Pang combination discloses the method of claim 1 and further discloses that one data type included among the plurality of disparate data types includes a sensor data type (Davidson, par. 24 discloses that the images 120 “may have been created using fluorescence imagery in which a specimen is dyed with fluorescent dye and excited with a light source,” such that the image data is generated by an optical sensor detecting the fluorescent emission).
Claim 3
Regarding claim 3, the Davidson/Pang combination discloses the method of claim 1, and further discloses that the coordinate system is an x-y-z coordinate system (Davidson discloses that, for a three-dimensional visual representation, “the X axis coordinate is based on a first dimension ... the Y axis coordinate is based on a second dimension, [and] the Z axis coordinate is based on a third dimension” of the dimensionally-reduced dataset).
Claim 4
Regarding claim 4, the Davidson/Pang combination discloses the method of claim 1, and further discloses that dynamically modifying the perspective views of the plot include three-dimensional rotations of the plot (Pang, col. 9, ll. 38-40 discloses that its object-detection algorithm “may search for both viewpoint rotation and in-plane rotation to achieve rotational invariance,” i.e., the point cloud is rotated in three dimensions to generate the plurality of perspective views that are analyzed). The motivation to combine Pang with Davidson is the same as set forth above with respect to claim 1.
Claim 7
Regarding claim 7, the Davidson/Pang combination discloses the method of claim 1, and further discloses that dynamically modifying the perspective views of the plot includes following a pre-programmed perspective modification trajectory (Pang discloses that its algorithm systematically “search[es] for both viewpoint rotation and in-plane rotation,” (Pang, col. 9, ll. 38-40) i.e., the point cloud is projected from a pre-determined, programmatically-defined sequence of viewpoints, rather than from viewpoints selected ad hoc by a user). The motivation to combine is the same as set forth above with respect to claim 1.
Claim 9
Regarding claim 9, the Davidson/Pang combination discloses the system of claim 13, and further discloses that the coordinate system is a three-dimensional coordinate system that is visible on a display (Davidson discloses displaying the three-dimensional visual representation 400 in a first region 510 of graphical user interface 500 on a display screen, e.g., “a monitor, a laptop computer display, [or] a tablet computer display,” coupled to the computer system).
Claim 10
Regarding claim 10, the Davidson/Pang combination discloses the system of claim 13, and further discloses that the particular feature has a shape that is recognizable by the computer vision algorithm (Pang discloses an “object detection algorithm” that detects a “target object” represented by a recognizable shape formed by a grouping of points within the projected two-dimensional depth images). The motivation to combine is the same as set forth above with respect to claim 1.
Claim 11
Regarding claim 11, the Davidson/Pang combination discloses the method of claim 1, and further discloses that the original set of data is a multi-modal multi-dimensional set of data (Davidson discloses that the images 120 include “a plurality of multispectral images of cells, a plurality of multimodal images of the cells, or both,” output as a “multi-dimensional dataset” of, e.g., 2,048 features per image channel).
Claim 12
Regarding claim 12, the Davidson/Pang combination discloses the method of claim 1, and further discloses that the plot is a three-dimensional scatter plot (Davidson discloses the three-dimensional embodiment of visual representation 400, in which icons 404 representing datapoints are positioned according to x, y, and z coordinates derived from the dimensionally-reduced dataset, which constitutes a three-dimensional scatter plot).
Claim 13
Regarding claim 13, Davidson discloses a computer system comprising a processor system and a storage system that stores instructions executable by the processor system (Davidson discloses “computer system 800” comprising “processor subsystem 880” coupled to “system memory 820,” wherein system memory 820 stores “program instructions executable by processor subsystem 880 to cause system 800 [to] perform various operations”).
The remaining limitations of claim 13 — accessing an original set of data having a plurality of disparate data types; generating a dimensionally reduced data set that is plottable in a coordinate system; plotting the dimensionally reduced data set to generate a visual plot; the one or more features including multiple portions of the dimensionally reduced data set clustered together to form the one or more features based on the perspective views of the plot being dynamically modified; in response to identifying a particular feature as viewed from a particular perspective view of the plot, sampling the original set of data to identify a data relationship; and triggering a re-training of the computer vision algorithm based on the identified data relationship — are disclosed by Davidson in combination with Pang for the same reasons, and are supported by the same citations, set forth above with respect to claim 1, applied here to the system context. In particular, Davidson’s express use of the term “clustering algorithm 106” to describe the grouping of datapoints discloses the “clustered together” language of claim 13.
As explained in the rejection of claim 1, Davidson does not disclose dynamically modifying perspective views of the plot in an attempt to identify one or more features that are visually detectable via a computer vision algorithm. As set forth above with respect to claim 1, Pang discloses generating a plurality of dynamically modified perspective views of three-dimensionally arranged data and applying a computer vision (object detection) algorithm to those views to identify a feature formed by a grouping of points depending on the viewpoint selected. Therefore, for the same reasons and with the same motivation to combine set forth above with respect to claim 1, it would have been obvious to a PHOSITA before the effective filing date to incorporate Pang’s perspective-view generation and computer-vision feature-detection technique into the object detection model visual analysis system of Davidson.
Claim 14
Regarding claim 14, the Davidson/Pang combination discloses the system of claim 13, and further discloses that a photographic snapshot of the feature at the particular perspective view of the plot is generated, and wherein the photographic snapshot is transmitted to a user for further review (Davidson discloses that selection of an icon representing a cluster/feature “cause[s] the display of the images 120 associated with the one or more particular datapoints” in a second region of the user interface, transmitting the underlying photographic image data associated with the identified feature to the user for review and labeling).
Claim 15
Regarding claim 15, the Davidson/Pang combination discloses the system of claim 13, and further discloses that the plot is a point cloud type of plot (Pang discloses receiving and processing a “three-dimensional (3D) point cloud,” the datapoints of which are projected into multiple two-dimensional depth images corresponding to the dynamically modified perspective views). The motivation to combine is the same as set forth above with respect to claim 1.
Claim 18
Regarding claim 18, the Davidson/Pang combination discloses the system of claim 13, and further discloses that the original set of data includes model behavior data (Davidson discloses that “second machine learning algorithm 110” outputs “predicted classification labels for unlabeled images 120,” and that these predicted labels — which reflect the behavior/output of a machine learning model — are factored back into dimensionality reduction algorithm 104 together with the image-derived features to generate an updated dimensionally-reduced dataset).
Claim 19
Regarding claim 19, Davidson discloses a method comprising accessing an original set of data having a plurality of disparate data types; generating a dimensionally reduced data set by performing a dimensional reduction operation on the original set of data, wherein the dimensionally reduced data set is plottable in a coordinate system; plotting the dimensionally reduced data set in the coordinate system, resulting in generation of a visual plot; wherein the one or more features include multiple portions of the dimensionally reduced data set grouped together to form the one or more features based on the perspective views of the plot being dynamically modified; and in response to a particular feature being identified, sampling the original set of data to identify a data relationship that exists within the original set of data, said data relationship being one that contributed to the feature being detectable, for the same reasons and supported by the same citations set forth above with respect to claim 1.
Davidson does not expressly disclose dynamically modifying perspective views of the plot in an attempt to identify the one or more features that are visually detectable. As set forth above with respect to claim 1, Pang, in the same field of endeavor, discloses generating a plurality of dynamically modified perspective views of three-dimensionally plotted data in an attempt to visually identify a feature formed by a grouping of datapoints depending on the viewpoint selected. Therefore, for the same reasons and with the same motivation to combine set forth above with respect to claim 1, it would have been obvious to a PHOSITA before the effective filing date to incorporate Pang’s dynamic perspective-view generation technique into Davidson’s visual analysis platform.
Claim 20
Regarding claim 20, the Davidson/Pang combination discloses the method of claim 19, and further discloses that identifying the particular feature is performed using a computer vision algorithm (Pang discloses that the “object detection algorithm” — a computer vision algorithm — detects the second plurality of 2D depth images (representing the target object) within the first plurality of 2D depth images (representing the scene), resulting in a plurality of 2D detection locations that are re-projected into 3D space). The motivation to combine is the same as set forth above with respect to claim 1.
Claims 5 are rejected under 35 U.S.C. 103 as being unpatentable over Davidson in view of Pang as applied to claim 1 above, and further in view of Takahashi (Pat. No. US 5,309,550).
Claim 5
Regarding claim 5, the Davidson/Pang combination discloses the method of claim 1, but neither Davidson nor Pang expressly discloses that dynamically modifying the perspective views of the plot includes bisecting the plot.
In the same field of endeavor of three-dimensional data visualization, Takahashi discloses generating a cross-sectional image of a three-dimensional object by storing “cutting plane data representing a cutting plane” and generating a surface image “in which the object is cut at the cutting plane,” i.e., bisecting the displayed three-dimensional structure to reveal interior features not otherwise visible from an external viewpoint (See the abstract). Therefore, it would have been obvious to a PHOSITA before the effective filing date to incorporate Takahashi’s cutting-plane/bisection technique into the dynamically modified perspective views of the Davidson/Pang combination, because bisecting a three-dimensionally plotted dataset to expose internal clusters is one of a finite number of predictable perspective-modification techniques (along with rotation and zoom) known in the data-visualization art, and a PHOSITA would have had a reasonable expectation that applying this known technique would successfully reveal internal clusters/features occluded by outer datapoints in Davidson’s three-dimensional plot.
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Davidson in view of Pang as applied to respective claims 1 and 13 above, and further in view of Werline (Pat. No. US 9,035,944).
Claim 6
Regarding claim 6, the Davidson/Pang combination discloses the method of claim 1, but neither Davidson nor Pang expressly discloses that dynamically modifying the perspective views of the plot includes a zooming operation.
In the same field of endeavor of interactive three-dimensional model manipulation, Werline discloses that a user “may allow a user to zoom-in or zoom-out from the model” (col. 2, ll. 62-64) independent of rotating or panning the displayed three-dimensional model. Therefore, it would have been obvious to a PHOSITA before the effective filing date to incorporate Werline’s zoom functionality into the dynamically modified perspective views of the Davidson/Pang combination as a routine and predictable addition to the known suite of view-manipulation controls (rotate, pan, zoom), to allow closer visual inspection of densely packed clusters identified in Davidson’s plot.
Claim 16
Regarding claim 16, the Davidson/Pang combination discloses the system of claim 13, but neither Davidson nor Pang expressly discloses that an axis rotation point is set within the plot, and wherein the axis rotation point changes.
In the same field of endeavor of interactive three-dimensional model manipulation, Werline discloses a “view cube” manipulator that allows a user to change the apparent orientation of a displayed three-dimensional model by rotating the model about one or more selectable axes, with the vantage point and effective rotation reference changing as different faces or quadrants of the view cube are activated. Therefore, it would have been obvious to a PHOSITA before the effective filing date to incorporate Werline’s changeable axis-of-rotation functionality into the dynamically modified perspective views of the Davidson/Pang combination, to give the user (or an automated routine) more precise, predictable control over which region of a three-dimensional plot is brought into view for computer-vision analysis, with a reasonable expectation of success given that this functionality was already known and in routine use in the analogous art of three-dimensional model manipulation.
Claims 8 are rejected under 35 U.S.C. 103 as being unpatentable over Davidson in view of Pang as applied to claim 1 above, and further in view of Tremblay et al. (Pub. No. US 2019/0355150).
Claim 8
Regarding claim 8, the Davidson/Pang combination discloses the method of claim 1, but neither Davidson nor Pang expressly discloses that dynamically modifying the perspective views is performed using a randomization factor.
In the same field of computer graphics, Tremblay teaches randomly determining the position and/or the orientation of a virtual camera with respect to a 3D scene (e.g., pan, tilt, and roll) via a synthetic training data generation system (see par. 61). As is well known in the art, the position and orientation of a virtual camera determines a perspective view of a virtual scene.
It would therefore have been obvious to a PHOSITA before the effective filing date to incorporate the teaching of Tremblay into the dynamically modified perspective views of the Davidson/Pang combination such that the perspective views would be dynamically modified using a randomization factor. The motivation would have been because a PHOSITA would have recognized only a finite number of predictable approaches for selecting the sequence of viewpoints to be searched — namely, a systematic (e.g., incremental or grid-based) search or a randomized search — such that selecting a randomization factor would have been obvious for yielding the predictable result of sampling the space of possible perspective views, with a reasonable expectation of success in locating visually detectable features.
Claims 17 are rejected under 35 U.S.C. 103 as being unpatentable over Davidson in view of Pang as applied to claim 13 above, and further in view of Griffin et al. (Pub. No. US 2004/0207625).
Claim 17
Regarding claim 17, the Davidson/Pang combination discloses the system of claim 13, but neither Davidson nor Pang expressly discloses that, while the perspective views of the plot are being dynamically modified, a filtering operation is performed on the plotted dimensionally reduced data set, said filtering operation removing noisy data in the plotted dimensionally reduced data set.
In the same field of data visualization, Griffin teaches that “Generally, use of a larger filter size is better at reducing large structured noise and is more sensitive to larger image features and larger motion, while use of a smaller filter size is more sensitive to smaller features and smaller motion,” (see pars. 443 and 648). Therefore, it would have been obvious to one skilled in the art before the effective filing date of the claimed invention to incorporate the noise-filtering technique taught by Griffin into the visualization system of the Davidson/Pang combination to improve the quality of rendered images.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHONG X NGUYEN whose telephone number is (571)270-1591. The examiner can normally be reached Mon-Fri 8am - 5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHONG X NGUYEN/ Primary Patent Examiner, Art Unit 2617
Prosecution Insights