DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/10/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Preliminary Amendment
The preliminary amendment filed 12/10/2024 has been acknowledged
Claims 1- 18 have been amended.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 4 recites the limitation " the optimization of the cost function of the distance between the two graphs". There is insufficient antecedent basis for this limitation in the claim.
Claim 5 recites “the optimization of the cost function of the distance between the two graphs” There is insufficient antecedent basis for this limitation in the claim.
Claim 11 recites “one class of the classifier” There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 7, 9, 10, 12-18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Thomas et al (Thomas hereinafter US 9996923 B2).
As per claim 1
Thomas teaches A computer-implemented method for generating at least one differential marker of the presence of a skin singularity of a human body, (Figure 1, Figure 4 Paragraph (50) “Where a correspondence is identified, such as in the cases of features F1 and F2, the system can display the images so that when a user selects a feature in one image, its corresponding appearance in the other image will be highlighted, for example, or provide some other suitable indication of their correspondence, such as a line connecting the appearances, indicia with matching alphanumeric information, shapes and/or colors, or an isolated or magnified view of the corresponding appearances, among other possibilities.”) said method comprising receiving at a first date of at least a first image of all or part of the human body (Figure 1, Paragraph (65) “The doctor captures a 3D image of a subject using a whole body system.”) forming a first part, of a first individual for displaying a dermoscopic image extracted from said first image with dermoscopic resolution (Figure 1, Figure 4, Paragraph (28) Using the estimated pose, a sub-section that corresponds to the right arm, for example, can be extracted from the two images and used in the subsequent models.) said first image comprising a plurality of cutaneous singularities of the skin of said body (Figure 1, Figure 4) each singularity having coordinates in a first reference frame associated with said first image (Figure 1, Paragraph (24) with two or more parameters at every pixel location of the image. Paragraph (33) “ Once the global motion model parameters have been estimated, the two images can then be aligned into a common global coordinate system and passed along to the next block.” Paragraph (35) “As with global scale motion estimation 230, intermediate scale motion estimation 240 can use image features to build a collection of correspondences between the image pairs….This sampling can be conducted using one of several possible techniques including, for example, grid-based sampling, Poisson disk sampling, stratified sampling, etc. ” Pixels have the coordinate descriptions of x and y. Each singularity is assigned pixels) and being associated with a first date (Paragraph (19) “comparing two or more images captured at different timepoints and/or using different imaging devices and/or modalities, such as for the purpose of building a spatial and/or temporal map of skin features in the images “ Figure 1, Paragraph (40) “Take for example, the task of estimating the observed change between two dermoscopy images of the same skin feature captured at two different points in time. In this case, the local scale motion might be sufficient to align the two images together and to track the change that is observed.” ) and at least a first value of a first descriptor (Paragraph (45) “Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc.”) each singularity located in the first image defining a node of a first graph (Figure 1, Figure 2, Figure 4, Paragraph (33) Once the global motion model parameters have been estimated, the two images can then be aligned into a common global coordinate system and passed along to the next block…” Paragraph (35) “As with global scale motion estimation 230, intermediate scale motion estimation 240 can use image features to build a collection of correspondences between the image pairs. … this sampling can be conducted using one of several possible techniques including, for example, grid-based sampling, Poisson disk sampling, stratified sampling, etc.” Paragraph (40) “In this case, the local scale motion might be sufficient to align the two images together and to track the change that is observed.” Paragraph (43) “In an exemplary implementation, the user selects for display a first one of the images in a pair of sequentially captured image…A segmentation procedure can then be used to identify skin features in the specified ROI.” Paragraph (44) “Spatio-temporal analytics block 320 then uses the parametric motion models determined as described above with reference to FIG. 2 to predict the location of the specified ROI and/or identified skin features in the second image of the image pair.” Paragraph (45) “) An additional feature detection 330 can be performed at the predicted ROI location in the second image to identify skin features of interest in the second image. Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc., the correspondence of said skin features can be established for the pair of images. Paragraph (68)” All the data would be aligned to one another so that the doctor could manually visualize temporal changes (color, texture, border, volume, etc.) from the data.” A “graph” is merely a visualization that shows data. A “node” is merely a piece of data. When the images are aligned in a grid like format it is essentially a graph. Each ROI and or the piexel(s) that are associated with a skin feature is effectively a node. Furthermore a segmented image in computer vision functions as a graph image where pixels or groups of pixels are treated as nodes and uses weighted edges to measure their similarity.) each node comprising attributes including a position of the singularity and at least one value of a descriptor (Paragraph (45) “Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc.”) receiving at a second date of at least one second image of the same first part of the human body of the first individual with substantially identical resolution, (Figure 1, Figure 4 Paragraph (7) “This disclosure also provides methods and apparatus to connect images of a subject, collected over multiple timepoints or..) Paragraph (19) “Another scenario involves comparing two or more images captured at different timepoints and/or using different imaging devices and/or modalities,”) said second image comprising a plurality of skin singularities of the skin of said body (Figure1, Figure 4) each singularity having coordinates in a second reference frame associated with said second image (Figure 1, Paragraph (24) with two or more parameters at every pixel location of the image. Paragraph (33) “ Once the global motion model parameters have been estimated, the two images can then be aligned into a common global coordinate system and passed along to the next block.” Paragraph (35) “As with global scale motion estimation 230, intermediate scale motion estimation 240 can use image features to build a collection of correspondences between the image pairs….This sampling can be conducted using one of several possible techniques including, for example, grid-based sampling, Poisson disk sampling, stratified sampling, etc. ” Pixels have the coordinate descriptions of x and y. Each singularity is assigned pixels) and being associated with a second date (Paragraph (19) “comparing two or more images captured at different timepoints and/or using different imaging devices and/or modalities, such as for the purpose of building a spatial and/or temporal map of skin features in the images “ Figure 1, Paragraph (40) “Take for example, the task of estimating the observed change between two dermoscopy images of the same skin feature captured at two different points in time. In this case, the local scale motion might be sufficient to align the two images together and to track the change that is observed.” ) and at least a second value of a first descriptor (Paragraph (45) “Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc.”) each singularity located in the second image defining a node of a second graph (Figure 1, Figure 2, Figure 4, Paragraph (33) Once the global motion model parameters have been estimated, the two images can then be aligned into a common global coordinate system and passed along to the next block…” Paragraph (35) “As with global scale motion estimation 230, intermediate scale motion estimation 240 can use image features to build a collection of correspondences between the image pairs. … this sampling can be conducted using one of several possible techniques including, for example, grid-based sampling, Poisson disk sampling, stratified sampling, etc.” Paragraph (36) “Intermediate scale motion estimation 240 can be carried out using one parametric model or multiple parametric models. As depicted in FIG. 2, intermediate scale motion estimation 240 can be decomposed into one or more components (240.1-240.M) that can be daisy-chained together to produce the composite set of parameters that describe the intermediate scale of motion.” Paragraph (40) “In this case, the local scale motion might be sufficient to align the two images together and to track the change that is observed.” Paragraph (43) “In an exemplary implementation, the user selects for display a first one of the images in a pair of sequentially captured image…A segmentation procedure can then be used to identify skin features in the specified ROI.” Paragraph (44) “Spatio-temporal analytics block 320 then uses the parametric motion models determined as described above with reference to FIG. 2 to predict the location of the specified ROI and/or identified skin features in the second image of the image pair.” Paragraph (45) “) An additional feature detection 330 can be performed at the predicted ROI location in the second image to identify skin features of interest in the second image. Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc., the correspondence of said skin features can be established for the pair of images.” Paragraph (68)” All the data would be aligned to one another so that the doctor could manually visualize temporal changes (color, texture, border, volume, etc.) from the data.” A “graph” is merely a visualization that shows data. A “node” is merely a piece of data. When the images are formed into a grid like format for tracking, either individual or aligned, it is essentially a graph. Each ROI and or the pixel(s) that are associated with a skin feature is effectively a node. Furthermore, a segmented image in computer vision functions as a graph image where pixels or groups of pixels are treated as nodes and uses weighted edges to measure their similarity. ) each node comprising attributes including a position of the singularity and at least one value of a descriptor (Paragraph (45) “Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc.”) a first representation comprising the first image and at least one first symbol associated with a first singularity located at a first position of said first image of the first reference frame (Figure 4, Paragraph (50) “features F1 and F2, the system can display the images so that when a user selects a feature in one image, its corresponding appearance in the other image will be highlighted, for example, or provide some other suitable indication of their correspondence, such as a line connecting the appearances, indicia with matching alphanumeric information, shapes and/or colors” ) said at least first symbol being superimposed on the first image (Figure 4, Paragraph (23) “In the aforementioned situations, exemplary embodiments can preferably receive user input to manually annotate/segment features or other areas of interest (AOI) in one or more of the images and have the system automatically transfer the annotation/segmentation across the images from the same capture set…” Paragraph (50) “features F1 and F2, the system can display the images so that when a user selects a feature in one image, its corresponding appearance in the other image will be highlighted, for example, or provide some other suitable indication of their correspondence, such as a line connecting the appearances, indicia with matching alphanumeric information, shapes and/or colors) at the first position, said first symbol having a first geometry and/or a first color generated as a function of at least the first value of the first descriptor considered at the first date (Figure 1, Figure 4 , “Paragraph (45) “Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc.”) “ Paragraph (50) “Where a correspondence is identified, such as in the cases of features F1 and F2, the system can display the images so that when a user selects a feature in one image, its corresponding appearance in the other image will be highlighted, for example, or provide some other suitable indication of their correspondence, such as a line connecting the appearances, indicia with matching alphanumeric information, shapes and/or colors, or an isolated or magnified view of the corresponding appearances’ Paragraph (51) “Where a correspondence is found but there appear to be differences between the two appearances of the feature, indicia can be generated in accordance with the detected degree of change so as to alert and/or assist the user in prioritizing the potential significance of detected features. For example, green indicia can be used for correspondences displaying little or no change, yellow for correspondences with an intermediate degree of change and red for correspondences with a significant degree of change. Metrics and other information can also be displayed.”) generating of a second representation in the vicinity of the first representation (Figure 1, Figure 4, Paragraph (41) “For example, the results can be presented to the user (e.g., doctor) via a display device or stored in a storage device for analysis at a later time. “ Paragraph (680 “Whenever required, the doctor could extract the collection of images for display, which could include the dermoscopy images, the 2D images from the 3D model, and possibly the 3D surface of the lesion. All the data would be aligned to one another so that the doctor could manually visualize temporal changes (color, texture, border, volume, etc.”) and at least one second symbol associated with the first singularity said second symbol having a second geometry and/or a second color (Figure 1, figure 2, “Paragraph (45) “Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc.”) “ Paragraph (50) “Where a correspondence is identified, such as in the cases of features F1 and F2, the system can display the images so that when a user selects a feature in one image, its corresponding appearance in the other image will be highlighted, for example, or provide some other suitable indication of their correspondence, such as a line connecting the appearances, indicia with matching alphanumeric information, shapes and/or colors, or an isolated or magnified view of the corresponding appearances’ Paragraph (51) “Where a correspondence is found but there appear to be differences between the two appearances of the feature, indicia can be generated in accordance with the detected degree of change so as to alert and/or assist the user in prioritizing the potential significance of detected features. For example, green indicia can be used for correspondences displaying little or no change, yellow for correspondences with an intermediate degree of change and red for correspondences with a significant degree of change. Metrics and other information can also be displayed.”) said at least one second symbol being superimposed on the second image at the first position (Figure1, Figure 4) , said second geometry and/or said second color being different from the first geometry and/or the first color thus defining a differential marker (Paragraph (51) Where a correspondence is found but there appear to be differences between the two appearances of the feature, indicia can be generated in accordance with the detected degree of change so as to alert and/or assist the user in prioritizing the potential significance of detected features. For example, green indicia can be used for correspondences displaying little or no change, yellow for correspondences with an intermediate degree of change and red for correspondences with a significant degree of change. Metrics and other information can also be displayed.” Paragraph (52) “Where a correspondence is missing, as in the cases of F3 and F5, indicia can be generated to indicate those feature appearances in one image lacking a corresponding appearance in the other image and/or indicia in or around the predicted locations of the missing appearances (e.g., P3 in Image 2 of FIG. 4).” ) when the calculated distance between a first value of the first descriptor calculated at the first date and a second value of the first descriptor calculated at the second date is greater than a predefined threshold (Paragraph (51) “Where a correspondence is found but there appear to be differences between the two appearances of the feature, indicia can be generated in accordance with the detected degree of change so as to alert and/or assist the user in prioritizing the potential significance of detected features. For example, green indicia can be used for correspondences displaying little or no change, yellow for correspondences with an intermediate degree of change and red for correspondences with a significant degree of change. Metrics and other information can also be displayed.” Paragraph (52) Where a correspondence is missing, as in the cases of F3 and F5, indicia can be generated to indicate those feature appearances in one image lacking a corresponding appearance in the other image and/or indicia in or around the predicted locations of the missing appearances (e.g., P3 in Image 2 of FIG. 4).Paragraph (59) “Preferably, the system can also generate and display statistics related to changes in one or more of the color, intensity, shape, border, size or texture of a skin feature, which could help the user analyze the feature from its onset to its termination” The change in value between the markers date is compared with some a threshold in order to determine severity determined by color.) the two images of each representation being oriented and aligned with each other by means of a step of comparing the two graphs and minimizing the error in the positional deviation of the nodes from each other. (Figure 2, Paragraph (32) “To estimate the parameters, global scale motion estimation 230 can use a set of correspondences obtained by comparing spatial/spectral image features (e.g., gradients, corners, oriented histograms, Fourier coefficients, etc.) Outliers from the correspondences thus obtained are preferably filtered out, such as by using a robust scheme like RANSAC or LMedS. “ Paragraph (33) “ Once the global motion model parameters have been estimated, the two images can then be aligned into a common global coordinate system and passed along to the next block.” Paragraph (35) As with global scale motion estimation 230, intermediate scale motion estimation 240 can use image features to build a collection of correspondences between the image pairs. In an exemplary implementation, the correspondence set is reduced to the inliers using an outlier rejection procedure derived from RANSAC…This produces a set of outlier-free sparse correspondences… sampling can be conducted using one of several possible techniques including, for example, grid-based sampling, Poisson disk sampling, stratified sampling, etc.” Paragraph (37) “As depicted in FIG. 2, intermediate scale motion estimation 240 can be decomposed into one or more components (240.1-240.M) that can be daisy-chained together to produce the composite set of parameters that describe the intermediate scale of motion.” Paragraph (40) “the local scale motion might be sufficient to align the two images together and to track the change that is observed.”)
As per claim 7
Thomas teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Thomas teaches wherein each singularity of the first image and/or of the second image is associated with a plurality of descriptors comprising at least one descriptor from the following list: a contrast value with respect to a value representative of an average color considered in the vicinity of the skin singularity;- a given class of a classifier of a neural network output having been trained with dermoscopic images of skin singularities ;- a characterization of a geometric shape datum,- a score corresponds to a scalar value or a numerical value obtained by implementing an algorithm processing as input an image extracted from the first image or the second image,- a score obtained by calculating different values of singularity descriptors considered in the vicinity of a given singularity (Paragraph (52) “Where a correspondence is identified, such as in the cases of features F1 and F2, the system can display the images so that when a user selects a feature in one image, its corresponding appearance in the other image will be highlighted, for example, or provide some other suitable indication of their correspondence, such as a line connecting the appearances, indicia with matching alphanumeric information, shapes and/or colors, or an isolated or magnified view of the corresponding appearances, among other possibilities” Paragraph (59) “preferably, the system can also generate and display statistics related to changes in one or more of the color, intensity, shape, border, size or texture of a skin feature, which could help the user analyze the feature from its onset to its termination.” Paragraph (68) “All the data would be aligned to one another so that the doctor could manually visualize temporal changes (color, texture, border, volume, etc.) from the data. As a diagnostic aid for the doctor, the system might also be able to automatically highlight changes inside the lesion (color or textural segmentation), and/or extrapolate the changes to a future possible time. “)
As per claim 9
Thomas teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Thomas teaches wherein an evolution criterion is calculated quantifying the evolution of a descriptor of a singularity between two images of two acquisitions made at two different dates (Paragraph (7) “The present disclosure sets out methods and apparatus for tracking changes that occur in skin features as the skin features evolve over time.” Paragraph (51) “Where a correspondence is found but there appear to be differences between the two appearances of the feature, indicia can be generated in accordance with the detected degree of change so as to alert and/or assist the user in prioritizing the potential significance of detected features. For example, green indicia can be used for correspondences displaying little or no change, yellow for correspondences with an intermediate degree of change and red for correspondences with a significant degree of change. Metrics and other information can also be displayed.” Paragraph (68) “Based on the temporal changes in lesion statistics (both intra-lesion changes and changes observed in neighboring lesions), specific lesions can be brought to the attention of the doctor who could then use dermoscopy to continue monitoring the evolving lesions…As a diagnostic aid for the doctor, the system might also be able to automatically highlight changes inside the lesion (color or textural segmentation), and/or extrapolate the changes to a future possible time. For example, the system could measure the area of a lesion over time, and predict its area at a future point using model-fitting strategies.” The evolution criterion calculated here is the change in appearance such as color or textural segmentation)
As per claim 10
Thomas teaches all claim limitations previously rejected in claim 9’s 102 rejection. See claim 9’s 102 rejection.
Thomas teaches wherein an evolution criterion is calculated from a distance defined between a first value of a descriptor of a first node of a first graph acquired at a first date and a second value of a descriptor of a second node of a second graph acquired at a second date (Figure 1, Figure 3, Figure 4, Figure 5A, Figure 5B, Paragraph (50) “Where a correspondence is identified, such as in the cases of features F1 and F2, the system can display the images so that when a user selects a feature in one image, its corresponding appearance in the other image will be highlighted, for example, or provide some other suitable indication of their correspondence, such as a line connecting the appearances, indicia with matching alphanumeric information, shapes and/or colors, or an isolated or magnified view of the corresponding appearances, among other possibilities” Paragraph (51) “Where a correspondence is found but there appear to be differences between the two appearances of the feature, indicia can be generated in accordance with the detected degree of change so as to alert and/or assist the user in prioritizing the potential significance of detected features. For example, green indicia can be used for correspondences displaying little or no change, yellow for correspondences with an intermediate degree of change and red for correspondences with a significant degree of change. Metrics and other information can also be displayed.” Paragraph (59) “ Preferably, the system can also generate and display statistics related to changes in one or more of the color, intensity, shape, border, size or texture of a skin feature, which could help the user analyze the feature from its onset to its termination.” The detected degree of change between corresponding descriptor values constitutes a distance between the values because it quantitatively represents the difference between the descriptor associated with the skin feature at the first date and the descriptor associated with the corresponding skin feature at the second date. ) each graph being generated from a first image, respectively a second image (Figure 4, Paragraph (40) “estimating the observed change between two dermoscopy images of the same skin feature captured at two different points in time.” See claim 1 mapping of skin features as respective graph nodes) said images corresponding to a body of the same individual and the first node and the second node having the same position within the first and second image. (Figure 1 and Figure 4, Paragraph (33) “Once the global motion model parameters have been estimated, the two images can then be aligned into a common global coordinate system and passed along to the next block.” Paragraph (44) “predict the location of the specified ROI and/or identified skin features in the second image of the image pair.” Paragraph (45) “Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc., the correspondence of said skin features can be established for the pair of images.”)
As per claim 12
Thomas teaches all claim limitations previously rejected in claim 1s 102 rejection. See claim 1’s 102 rejection.
Thomas teaches wherein a third symbol is generated according to a given color and/or shape (Paragraph (50) “Where a correspondence is identified, such as in the cases of features F1 and F2, the system can display the images so that when a user selects a feature in one image, its corresponding appearance in the other image will be highlighted, for example, or provide some other suitable indication of their correspondence, such as a line connecting the appearances, indicia with matching alphanumeric information, shapes and/or colors, or an isolated or magnified view of the corresponding appearances, among other possibilities.” when a singularity is present in a first image acquired at a given position for the first time, (Figure 4 Paragraph (49) “With respect to 570, a correspondence can be deemed to be missing if no skin feature is detected at a location predicted at 530, such as in the case of feature F3 shown in FIG. 4, or if a feature is detected in image 2, at a location which was not predicted at 530, such as in the case of feature F5 Where a feature appears in one image and not the other, the expected location of the feature in the image in which it is missing can be predicted from the location of the feature in the image in which it appears.”) said color or shape of the third symbol (Paragraph (50)” indicia with matching alphanumeric information, shapes and/or colors, or an isolated or magnified view of the corresponding appearances, among other possibilities” Paragraph (52) “Where a correspondence is missing, as in the cases of F3 and F5, indicia can be generated to indicate those feature appearances in one image lacking a corresponding appearance in the other image and/or indicia in or around the predicted locations of the missing appearances (e.g., P3 in Image 2 of FIG. 4) Under broadest reasonable interpretation Thomas shows using distinguishable visual indicia to show different feature correspondence states.) enabling said symbol to be distinguished from mother symbol to indicate the new appearance of said singularity. (Figure 4, Paragraph (49) “With respect to 570, a correspondence can be deemed to be missing if no skin feature is detected at a location predicted at 530, such as in the case of feature F3 shown in FIG. 4, or if a feature is detected in image 2, at a location which was not predicted at 530” Paragraph (52) “Where a correspondence is missing, as in the cases of F3 and F5, indicia can be generated to indicate those feature appearances in one image lacking a corresponding appearance in the other image and/or indicia in or around the predicted locations of the missing appearances (e.g., P3 in Image 2 of FIG. 4).” Thomas’s system generates indications of correspondence relationships, identifying the relationships as unchanged, missing or changed”)
As per claim 13
Thomas teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Thomas teaches wherein user interaction with at least one displayed symbol (Figure 5A Paragraph (50) “Where a correspondence is identified, such as in the cases of features F1 and F2, the system can display the images so that when a user selects a feature in one image”) generates a first digital instruction (Figure 6) for displaying at least one dermoscopic image (Paragraph (50) “its corresponding appearance in the other image will be highlighted…or an isolated or magnified view of the corresponding appearances, among other possibilities.“ Paragraph (54) or an isolated or magnified view of the corresponding appearances, among other possibilities…at 521, receives user input identifying individual skin feature(s) in the first image (in which case, the specification of an ROI can be omitted); at 530, predicts the location(s) in the second image of the skin feature(s) detected in the first image; at 551, receives user input identifying individual skin feature(s) in the second image; at 560 identifies any correspondence(s) unchanged, missing, and/or changed; and at 570 generates indications of such correspondence relationships (e.g., unchanged, missing, changed), as described above.” Paragraph (55) “to assist the user with this step, the system displays in or on the second image the location(s) predicted in step 530 of the skin feature(s) identified by the user in the first image”) Paragraph (43) “the user selects for display a first one of the images in a pair of sequentially captured images. Preferably, the system is configured to allow the user to select either the earlier or later captured image of the pair…user interface (GUI), may specify on the selected image a region of interest (ROI) within which features are to be detected.” Paragraph (44) “Spatio-temporal analytics block 320 then uses the parametric motion models determined as described above with reference to FIG. 2 to predict the location of the specified ROI and/or identified skin features in the second image of the image pair.” ) in a display window (Paragraph (43) “ the user, such as with the use of a suitable graphical user interface (GUI),” ) said displayed dermoscopic image corresponding to an image extracted from the first image (Figure 4, Figure 5A, Paragraph (43) “may specify on the selected image a region of interest (ROI) within which features are to be detected” Paragraph (45) “the user can identify the skin features in the first and/or second image, such as by use of the GUI.” Paragraph (48) “ receives input from the user specifying a ROI in the first image of the image pair”) associated with the position at which the symbol is displayed on the first image (Figure 4 Paragraph 48 as depicted in the flow chart of FIG. 5A and with reference to FIG. 4, in an exemplary procedure in accordance with the arrangement 300 of FIG. 3, an exemplary system: at 510, receives input from the user specifying a ROI in the first image of the image pair; at 520 detects skin feature(s) (e.g., F1-F4) within the ROI in the first image; at 530 predicts the location(s) in the second image of the skin feature(s) detected in the first image; at 540 predicts the location in the second image of the ROI specified in the first image; at 550 detects skin feature(s) (e.g., F1′, F2′, F4′) within the predicted ROI' in the second image; at 560 identifies any correspondence(s) between the skin feature(s) detected in the first image and the skin feature(s) detected in the second image, such as by a) comparing the predicted location(s) from step 530 with the location(s) of skin feature(s) detected in step 550, or by b) comparing the spatial arrangement of skin feature(s) detected in step 520 within the ROI in the first image to the spatial arrangement of skin feature(s) detected in step 550 within the ROI' in the second image; and at 570 generates an indication of any correspondence(s) identified (e.g., F1-F1′, F2-F2′, F4-F4′) and/or an indication if any expected correspondence is missing in either direction (Image 1 to Image 2 or vice versa), so as to bring this to the user's attention. “)
As per claim 14
Thomas teaches all claim limitations previously rejected under claim 1’s 102 rejection. See claim 1’s 102 rejection.
Thomas teaches wherein a second digital instruction (Figure 6) generated by a user action enables two dermoscopic images to be displayed side by side (Figure 1 Figure 4 Paragraph (55) “to assist the user with this step, the system displays in or on the second image the location(s) predicted in step 530 of the skin feature(s) identified by the user in the first image. For example, as shown in FIG. 4, P3 indicates the location in Image 2 predicted in step 3 for feature F3.”) extracted respectively from a first image and from a second image (Figure 4) said two dermoscopic images enabling the singularities of the same position on the body to be displayed at the same resolution and on the same dimensional scale. (Figure 1 , Figure 4, Paragraph (33) “Once the global motion model parameters have been estimated, the two images can then be aligned into a common global coordinate system and passed along to the next block.” Paragraph (68) “All the data would be aligned to one another so that the doctor could manually visualize temporal changes” )
As per claim 15
Thomas teaches all claim limitations previously rejected under claim 1’s 102 rejection. See claim 1’s 102 rejection.
Thomas teaches wherein a first digital command (Figure 6) for moving, zooming or electing an area of interest in the first image of the first representation automatically generates an identical digital command for an equivalent area of interest in the second image of the second representation (Paragraph (50) “Where a correspondence is identified, such as in the cases of features F1 and F2, the system can display the images so that when a user selects a feature in one image, its corresponding appearance in the other image will be highlighted, for example, or provide some other suitable indication of their correspondence, such as a line connecting the appearances, indicia with matching alphanumeric information, shapes and/or colors, or an isolated or magnified view of the corresponding appearances, among other possibilities.”)
As per claim 16
Thomas teaches all claim limitations previously rejected under claim 1’s 102 rejection. See claim 1’s 102 rejection.
Thomas teaches wherein a second numerical control (Figure 6) enables a three- dimensional digital avatar of an individual's body to be oriented so as to display a portion of the body (Figure 1, Paragraph (29) “Pose estimation 220 would typically only be required in conjunction with input images that cover a large field of view, such as those captured with a 3D total body imaging device” Paragraph (41) “mage of a skin feature to a 3D whole body image of the same subject for the purposes of automatically identifying and tagging the dermoscopy image to the corresponding skin feature on the 3D whole body image.” Paragraph (65) “he lesions captured with the dermatoscope are compared with the lesions detected in the 3D model, and automatically aligned and tagged with the lesions on the 3D model that provide for the best possible match “) a third numerical control enabling the said portion of the body displayed to be magnified over an area of interest (Paragraph (50) “Where a correspondence is identified, such as in the cases of features F1 and F2, the system can display the images so that when a user selects a feature in one image, its corresponding appearance in the other image will be highlighted, for example, or provide some other suitable indication of their correspondence, such as a line connecting the appearances, indicia with matching alphanumeric information, shapes and/or colors, or an isolated or magnified view of the corresponding appearances, among other possibilities.” said area of interest displaying a plurality of markers each having a position on the surface of the human body in a reference frame associated with the digital avatar each marker being associated with a singularity of the human body, (Figure 1, Figure 4, Paragraph (65) “The doctor captures a 3D image of a subject using a whole body system. During the same visit, the doctor also uses a dermatoscope to capture cross-polarized images of lesions on the subject. The lesions captured with the dermatoscope are compared with the lesions detected in the 3D model, and automatically aligned and tagged with the lesions on the 3D model that provide for the best possible match (auto-tagging)” Paragraph (68) “ the subject is captured using the 3D imaging system and correspondences are established between lesions across all the available visits. Based on the temporal changes in lesion statistics (both intra-lesion changes and changes observed in neighboring lesions), specific lesions can be brought to the attention of the doctor who could then use dermoscopy to continue monitoring the evolving lesions” ) a fourth digital command for electing said marker to display a dermoscopic image extracted from the first image said extracted image being defined around the position of the selected marker (Figure 1, Figure 4, Paragraph (43) “the user selects for display a first one of the images in a pair of sequentially captured images. Preferably, the system is configured to allow the user to select either the earlier or later captured image of the pair. Feature detection 310 is then performed on the selected image. In exemplary embodiments, the user, such as with the use of a suitable graphical user interface (GUI), may specify on the selected image a region of interest (ROI) within which features are to be detected. In exemplary embodiments, the system can automatically or semi-automatically generate the ROI, such as, for example, an ROI of a selected anatomical region, such as the skin of the face. The specification of an ROI may be desirable where the selected image covers a large area or a complex area, such as the face, or includes more features than those in which the user is interested. A segmentation procedure can then be used to identify skin features in the specified ROI.” Paragraph (50) “Where a correspondence is identified, such as in the cases of features F1 and F2, the system can display the images so that when a user selects a feature in one image, its corresponding appearance in the other image will be highlighted, for example, or provide some other suitable indication of their correspondence, such as a line connecting the appearances, indicia with matching alphanumeric information, shapes and/or colors, or an isolated or magnified view of the corresponding appearances, among other possibilities.”)
As per claim 17
Thomas teaches all claim limitations previously rejected under claim 1’s 102 rejection. See claim 1’s 102 rejection.
Thomas teaches wherein the dermoscopic images are acquired by an image- taking device configured to acquire a plurality of images of the skin of a human body of an individual and to assign to each image a position on a 3D model representing the body of said individual. (Figure 1, Paragraph (41) “image of a skin feature to a 3D whole body image of the same subject for the purposes of automatically identifying and tagging the dermoscopy image to the corresponding skin feature on the 3D whole body image. “ Paragraph (59) “Preferably, this temporal data sequence (image and change statistics) can be coupled with a 3D system to provide topographical changes and/or with high-resolution dermoscopy for detailed textural changes, both of which would provide additional information helpful to the user.” Paragraph (65) “The doctor captures a 3D image of a subject using a whole body system. During the same visit, the doctor also uses a dermatoscope to capture cross-polarized images of lesions on the subject. The lesions captured with the dermatoscope are compared with the lesions detected in the 3D model, and automatically aligned and tagged with the lesions on the 3D model that provide for the best possible match (auto-tagging).” Paragraph (68) “e imaging modality, type of device used, and capture timepoints change. The doctor uses a 3D imaging system (e.g., Canfield Scientific Inc.'s VECTRA or WB360 system) to monitor a subject over multiple visits. During each visit, the subject is captured using the 3D imaging system and correspondences are established between lesions across all the available visits…Moving forward, each time a cross-polarized dermoscopy image of a lesion is captured, it is auto-tagged to a lesion on the 3D system, and appended to the collection of images of the lesion under consideration. Whenever required, the doctor could extract the collection of images for display, which could include the dermoscopy images, the 2D images from the 3D model, and possibly the 3D surface of the lesion. Paragraph (69) “ 3D human body imaging devices (e.g., Canfield Scientific Inc.'s VECTRA), and/or 3D Total Body systems (e.g., Canfield Scientific Inc.'s WB360), 3D volumetric imaging devices, among others.” )
As per claim 18
Thomas teaches all claim limitations previously rejected under claim 1’s 102 rejection. See claim 1’s 102 rejection.
Thomas teaches A system comprising an electronic terminal including a display for generating images produced by the method of claim 1 and a data exchange interface for receiving images acquired by an image acquisition device. (Figure 6, Paragraph (43) “In exemplary embodiments, the user, such as with the use of a suitable graphical user interface (GUI),” Paragraph (73) “ Processing module 140 may be coupled to storage 150, for storing and retrieving images and motion models, among other data, and to input/output devices 160, such as a display device and/or user input devices,”)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-5, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Thomas et al (Thomas hereinafter US 9996923 B2) in view of Rahman et al (Rahman hereinafter US 20210118550 A1)
As per claim 2
Thomas teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Thomas does not teach wherein at least one feature vector is calculated at each node of the first graph and of the second graph by a machine learning model said model receiving as input an image of a singularity and generating as output a feature vector of the similarity of said image.
Rahman teaches wherein at least one feature vector is calculated at each node of the first graph and of the second graph by a machine learning model (Figure 3, Figure 4 Paragraph [0029] “ Using the transfer learning approach, the deep features of the confirmed images are extracted by passing them through the CNNs that are without a classification head.”) said model receiving as input an image of a singularity and generating as output a feature vector of the similarity of said image (Figure 5, Figure 4, Figure 3 Paragraph [0034] “The output of the feature learning phase is a deep feature vector that is passed to classification module 150. The output of the CNN architecture is thus the feature vector learned during the feature learning stage.” Paragraph [0038] “we may actually take these 7×7×512=25,088 values and treat them as a feature vector that quantifies the contents of an image.” Paragraph [0044] “corr(Xu, Yv) thus provides a single, combined feature vector that fuses features extracted from the subject image, and thus represents all extracted features in that single feature vector. Such feature extraction process is carried out on both pre-existing images of skin legions whose pathologies had already been established (for purposes of building database 170), and on query images captured by image “
Thomas’ skin feature singularity are the nodes of the first and second graph. Rahman is being relied upon for teaching the wat of characterizing the image associated with each skin singularity node. Rahman feeds the skin lesion image into a CNN and obtains a feature vector that quantifies the image.
Specifically, Rahman teaches applying deep feature extraction to a skin lesion image using said CNN to “generate a feature vector quantifying contents of the digital query image” Applying Rahman’s feature extraction methodology to Thomas results in the image associated with each skin node being processed by a machine learning model to generate a respective feature vector characterizing that node.
Accordingly, a person of ordinary skill in the art, at the time this invention was effectively filed would have found it obvious to modify Thomas’s methodology with Rahman’s concept of applying a CNN based feature extraction to the image associated with each skin node. A person of ordinary skill in the art would do this to generate feature vectors that quantify the visual features of each skin node. A person of ordinary skill in the art see’s the advantage of an improved identification and comparison of corresponding skin singularities by providing discriminating feature representations for determining similarity between skin features.
As per claim 3
Thomas and Rahman teach all claim limitations previously rejected in claim 2’s 103 rejection. See claim 2.
Rahman teaches The method according to claim 2, characterized in that wherein the comparison step implements the optimization of a cost function of the calculation of a distance between two graphs (Paragraph [0047] The difference between the feature vector of the query image (patient lesion) and the feature vectors of lesions of reference images in database 170 is preferably calculated based on different distance measures, such as Euclidean, Manhattan, and Cosine methods (which methods are known to those skilled in the art) to compute the similarity between the query image and the database… The smaller the difference (i.e., “distance”), the higher the computed “similarity” level is between the two compared ROIs. The searching and retrieval result of the CBIR algorithm depends on the effectiveness of the distance metrics to measure the similarity level among the selected images. Preferably, the query-specific adaptive similarity fusion approach set forth herein effectively exploits the online lesion classification information and adjusts the feature weights accordingly in a dynamic fashion.” Rahman’s distance metric functions as a cost function because it assigns quantitative cost of distance to the difference between two feature vectors wherein minimizing the calculated distance spotlights the feature vectors having greatest similarity. ) a first distance between the nodes of the first graph and the nodes of the second graph, said first distance using a geometric metric for calculating a distance between points in space (Paragraph [0044] “Distance measures are applied to the query features and the features from the database images from database based on the closeness of those features” Paragraph [0047] The difference between the feature vector of the query image (patient lesion) and the feature vectors of lesions of reference images in database 170 is preferably calculated based on different distance measures, such as Euclidean, Manhattan, and Cosine methods (which methods are known to those skilled in the art) to compute the similarity between the query image and the database) a second distance between the nodes of the first graph and the nodes of the second graph, said second distance using a metric for calculating a distance between feature vectors. (Paragraph [0044] “ Distance measures are applied to the query features and the features from the database images from database based on the closeness of those features” Paragraph [0047] The difference between the feature vector of the query image (patient lesion) and the feature vectors of lesions of reference images in database 170 is preferably calculated based on different distance measures, such as Euclidean, Manhattan, and Cosine methods (which methods are known to those skilled in the art) to compute the similarity between the query image and the database)
Thomas teaches a first distance between the nodes of the first graph and the nodes of the second graph, said first distance using a geometric metric for calculating a distance between points in space (Paragraph (32) “ global scale motion estimation 230 can use a set of correspondences obtained by comparing spatial/spectral image features (e.g., gradients, corners, oriented histograms, Fourier coefficients, etc.) Outliers from the correspondences thus obtained are preferably filtered out, such as by using a robust scheme like RANSAC or LMedS.” Paragraph (35) “As with global scale motion estimation 230, intermediate scale motion estimation 240 can use image features to build a collection of correspondences between the image pairs…this model estimates the weights assigned to each of the correspondences so as to compute the parametric transformation.” Paragraph After intermediate scale motion estimation 240, local scale motion estimation 250 estimates the finest scale of non-rigid deformation occurring between the pair of input images. The non-rigid deformation can be treated as being composed of a horizontal and a vertical displacement at every pixel in the image. This displacement could represent a linear displacement, such as a translation) Paragraph (44) “ Spatio-temporal analytics block 320 then uses the parametric motion models determined as described above with reference to FIG. 2 to predict the location of the specified ROI and/or identified skin features in the second image of the image pair.” Paragraph (45) “. Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc., the correspondence of said skin features can be established for the pair of images.” Thomas’s skin feature singularities correspond to the nodes of the respective graphs. Therefore, when Thomas determines correspondence using the spatial configuration, locations, horizontal and vertical displacement of those features teaches determining the geometric and spatial relationship between nodes of the two graph images.)
In a combined teaching Thomas establishes correspondence between skin feature singularity nodes of first and second images based on their spatial configuration and positional displacement. This also includes computing a transformation for aligning corresponding image locations. Rahman in turn teaches characterization of the image associated with each skin singularity using a CNN generated feature vector and calculating a distance between the feature vectors using Euclidean, Manhattan or Cosine distance. These evaluations allow smaller distance to represent a larger similarity. The combined teaching allows for comparison of the nodes of the two graphs taking into account both the geometric and spatial relationship between nodes and the distance between feature vectors characterizing specific nodes with the distance metric enabling a quantitative cost for determining the correspondence having the largest similarity.
As per claim 4
Thomas and Rahman teach all claim limitations previously rejected in claim 2’s 103 rejection. See claim 2.
Rahman teaches the optimization of the cost function (Paragraph [0047] The difference between the feature vector of the query image (patient lesion) and the feature vectors of lesions of reference images in database 170 is preferably calculated based on different distance measures, such as Euclidean, Manhattan, and Cosine methods (which methods are known to those skilled in the art) to compute the similarity between the query image and the database… The smaller the difference (i.e., “distance”), the higher the computed “similarity” level is between the two compared ROIs. The searching and retrieval result of the CBIR algorithm depends on the effectiveness of the distance metrics to measure the similarity level among the selected images. Preferably, the query-specific adaptive similarity fusion approach set forth herein effectively exploits the online lesion classification information and adjusts the feature weights accordingly in a dynamic fashion.” Rahman’s distance metric functions as a cost function because it assigns quantitative cost of distance to the difference between two feature vectors wherein minimizing the calculated distance spotlights the feature vectors having greatest similarity. Rahman showcases optimizing a quantitative distance to similarity determination between feature vectors. )
Thomas in view of Rahman teaches the distance between the two graphs. Thomas’s skin features outline the respective nodes of the first and second graphs. Thomas then cements correspondence between the two features across the two images Paragraph (35) “As with global scale motion estimation 230, intermediate scale motion estimation 240 can use image features to build a collection of correspondences between the image pairs….This sampling can be conducted using one of several possible techniques including, for example, grid-based sampling, Poisson disk sampling, stratified sampling, etc. ” Pixels have the coordinate descriptions of x and y. Each singularity is assigned pixels”, Paragraph (45) “Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc., the correspondence of said skin features can be established for the pair of images.”
Rahman the enables the quantitative distance between feature representations with those nodes within the modified system: Paragraph [0044] “Distance measures are applied to the query features and the features from the database images from database based on the closeness of those features” Paragraph [0047] The difference between the feature vector of the query image (patient lesion) and the feature vectors of lesions of reference images in database 170 is preferably calculated based on different distance measures, such as Euclidean, Manhattan, and Cosine methods (which methods are known to those skilled in the art) to compute the similarity between the query image and the database”.
When Rahman’s feature vector distance technique is applied to Thomas’ skin feature nodes, the resulting distances characterize differences between the respective nodes of Thomas’s first and second graphs.
Thomas teaches enables a transformation to be applied to each node of a first graph (Paragraph (32) “The model parameters can be subsequently computed using model fitting strategies”, Paragraph (33) “Once the global motion model parameters have been estimated, the two images can then be aligned into a common global coordinate system and passed along to the next block.”, Paragraph (35) “Using the sampled inlier correspondences, this model estimates the weights assigned to each of the correspondences so as to compute the parametric transformation.” This shows Thomas computes and applies a transformation to the corresponding image features. Furthermore, Thomas states that the transformation is at a pixel level: Paragraph (38) “The non-rigid deformation can be treated as being composed of a horizontal and a vertical displacement at every pixel in the image.” The parametric transformation and pixel level displacement show applying transformation to the nodes of the first graph) to make it correspond to a node of the second graph (Paragraph (44) “Spatio-temporal analytics block 320 then uses the parametric motion models determined as described above with reference to FIG. 2 to predict the location of the specified ROI and/or identified skin features in the second image of the image pair.” Here Thomas shows using the transformation to predict the corresponding skin feature to the second image. Thomas then gives the results: Paragraph (45) “An additional feature detection 330 can be performed at the predicted ROI location in the second image to identify skin features of interest in the second image. Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc., the correspondence of said skin features can be established for the pair of images.” This shows Thomas’ transformation of the first image to predict and create correspondence with the skin feature in the second image.
In a combined teaching, Rahman teaches using a quantitative feature vector distance as a cost for determining similarity where minimizing distance identifies the best similarity. Meanwhile Thomas teaches computing a parametric transformation from image feature correspondences and uses that transformation to predict and establish the linked skin feature location in the second image. Incorporating Rahman’s distance based characterization into Thomas’ correspondence methodology results in the cost/distance between the asserted graphs being used with creating the correspondences Thomas computes and applies the transformation between the nodes.
As per claim 5
Thomas and Rahman teach all claim limitations previously rejected under claim 2’s 103 rejection. See claim 2’s 103 rejection.
Rahman teaches the optimization of the cost function (Paragraph [0047] The difference between the feature vector of the query image (patient lesion) and the feature vectors of lesions of reference images in database 170 is preferably calculated based on different distance measures, such as Euclidean, Manhattan, and Cosine methods (which methods are known to those skilled in the art) to compute the similarity between the query image and the database… The smaller the difference (i.e., “distance”), the higher the computed “similarity” level is between the two compared ROIs. The searching and retrieval result of the CBIR algorithm depends on the effectiveness of the distance metrics to measure the similarity level among the selected images. Preferably, the query-specific adaptive similarity fusion approach set forth herein effectively exploits the online lesion classification information and adjusts the feature weights accordingly in a dynamic fashion.” Rahman’s distance metric functions as a cost function because it assigns quantitative cost of distance to the difference between two feature vectors wherein minimizing the calculated distance spotlights the feature vectors having greatest similarity. Rahman showcases optimizing a quantitative distance to similarity determination between feature vectors. )
Thomas in view of Rahman teaches the distance between the two graphs. Thomas’s skin features outline the respective nodes of the first and second graphs. Thomas then cements correspondence between the two features across the two images Paragraph (35) “As with global scale motion estimation 230, intermediate scale motion estimation 240 can use image features to build a collection of correspondences between the image pairs….This sampling can be conducted using one of several possible techniques including, for example, grid-based sampling, Poisson disk sampling, stratified sampling, etc. ” Pixels have the coordinate descriptions of x and y. Each singularity is assigned pixels”, Paragraph (45) “Based on the spatial configuration of the skin features within the two ROIs and by using pixel measurements such as border, color, intensity, texture, etc., the correspondence of said skin features can be established for the pair of images.”
Rahman the enables the quantitative distance between feature representations with those nodes within the modified system: Paragraph [0044] “Distance measures are applied to the query features and the features from the database images from database based on the closeness of those features” Paragraph [0047] The difference between the feature vector of the query image (patient lesion) and the feature vectors of lesions of reference images in database 170 is preferably calculated based on different distance measures, such as Euclidean, Manhattan, and Cosine methods (which methods are known to those skilled in the art) to compute the similarity between the query image and the database”.
When Rahman’s feature vector distance technique is applied to Thomas’ skin feature nodes, the resulting distances characterize differences between the respective nodes of Thomas’s first and second graphs. Rahman’s distance calculation provides a quantitative distance between he feature representations correlated with nodes of Thomas’ first and second graphs.
Thomas teaches enables a non-rigid transformation to be applied (Paragraph (38) “After intermediate scale motion estimation 240, local scale motion estimation 250 estimates the finest scale of non-rigid deformation occurring between the pair of input images. The non-rigid deformation can be treated as being composed of a horizontal and a vertical displacement at every pixel in the image.” Paragraph (40) “Take for example, the task of estimating the observed change between two dermoscopy images of the same skin feature captured at two different points in time. In this case, the local scale motion might be sufficient to align the two images together and to track the change that is observed.” Paragraph (40) connects non rigid transformation to two temporally separated dermoscopic images of the same skin feature.
In combination, Rahman enables the system to determine similarity between skin lesion feature representations using a feature vector distance while Thomas teaches aligning images of the corresponding skin features using local scale motion estimation employing a non-rigid deformation model. This includes horizontal and vertical displacement at a pixel level. Applying Rahman’s distance based characterization within Thomas’s methodology results in the distance/cost between the graphs being used to create correspondences while the non rigid transformation aligns the corresponding skin feature data.
As per claim 11
Thomas teaches all claim limitations previously rejected in claim 9’s 102 rejection. See claim 9’s 102 rejection.
Thomas teaches wherein the color and/or geometry of a symbol are selected according to: a descriptor value exceeding a threshold (Paragraph (51) “Where a correspondence is found but there appear to be differences between the two appearances of the feature, indicia can be generated in accordance with the detected degree of change so as to alert and/or assist the user in prioritizing the potential significance of detected features. For example, green indicia can be used for correspondences displaying little or no change, yellow for correspondences with an intermediate degree of change and red for correspondences with a significant degree of change. Metrics and other information can also be displayed.” Paragraph (52) Where a correspondence is missing, as in the cases of F3 and F5, indicia can be generated to indicate those feature appearances in one image lacking a corresponding appearance in the other image and/or indicia in or around the predicted locations of the missing appearances (e.g., P3 in Image 2 of FIG. 4).Paragraph (59) “Preferably, the system can also generate and display statistics related to changes in one or more of the color, intensity, shape, border, size or texture of a skin feature, which could help the user analyze the feature from its onset to its termination” The change in value between the markers date is compared with a threshold in order to determine severity determined by color.) ) the value of an evolution criterion for a singularity descriptor calculated between two first images acquired at two dates (Paragraph (7) “The present disclosure sets out methods and apparatus for tracking changes that occur in skin features as the skin features evolve over time.” Paragraph (51) “Where a correspondence is found but there appear to be differences between the two appearances of the feature, indicia can be generated in accordance with the detected degree of change so as to alert and/or assist the user in prioritizing the potential significance of detected features. For example, green indicia can be used for correspondences displaying little or no change, yellow for correspondences with an intermediate degree of change and red for correspondences with a significant degree of change. Metrics and other information can also be displayed.” Paragraph (68) “Based on the temporal changes in lesion statistics (both intra-lesion changes and changes observed in neighboring lesions), specific lesions can be brought to the attention of the doctor who could then use dermoscopy to continue monitoring the evolving lesions…As a diagnostic aid for the doctor, the system might also be able to automatically highlight changes inside the lesion (color or textural segmentation), and/or extrapolate the changes to a future possible time. For example, the system could measure the area of a lesion over time, and predict its area at a future point using model-fitting strategies.” The evolution criterion calculated here is the change in appearance such as color or textural segmentation. This criterion changes the color of the indicia)
In regards to “a criterion for a singularity to belong to at least one class of the classifier” In the combined Thomas/Rahman, Rahman teaches supplying an image of a skin singularity to a neural network based classifier to determine a class associated with the singularity while Thomas teaches visually identifying characteristics of a skin feature using indicia having different “shapes and/or colors”. It would have been obvious to the skilled artisan to use Thomas’s disclosed symbol color and or geometry to visually represent the class determined by Rahman’s classifier so that the color or geometry of the displayed symbol is selected according to the class assigned to the corresponding skin singularity. This modification simply uses Thomas’ disclosed visual indicia to communicate the result of Rahman’s automated classification of the same type of skin feature. Rahman determines the class; Thomas already teaches encoding the information about the skin feature into symbol color and shape. The combination uses Thomas’ visual encoding to display Rahman’s classification result.
Claims 6 is rejected under 35 U.S.C. 103 as being unpatentable over Thomas et al (Thomas hereinafter US 9996923 B2) in view of Rahman et al (Rahman hereinafter US 20210118550 A1) in further view of Patiño (Patiño hereinafter “Automatic skin lesion segmentation on dermoscopic images by the means of superpixel merging”)
As per claim 6
Thomas teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Thomas recognizes in paragraph (3) that “identifying, annotating, and tracking thousands of lesions across a number of patient visits is a daunting task, both in terms of effort and liability” Thomas states in paragraph (17) that “Skin images may be captured on any device that either captures the full human body or any subsection thereof” and that ROI can be selected to limit the skin features in question: Paragraph (43) “ the user, such as with the use of a suitable graphical user interface (GUI), may specify on the selected image a region of interest (ROI) within which features are to be detected. In exemplary embodiments, the system can automatically or semi-automatically generate the ROI, such as, for example, an ROI of a selected anatomical region, such as the skin of the face. The specification of an ROI may be desirable where the selected image covers a large area or a complex area,” This shows that the number of skin feature nodes processed is necessarily affected by the size and content of the selected ROI. A person of ordinary skill in the art would have found it an obvious selection of range when Thomas already contemplates thousands of lesions and expressly allows restricting processing to a selected ROI.
Nonetheless, Thomas does not explicitly denote a number of nodes within this claimed range. Therefore, Thomas does not teach wherein each graph comprises between 50 and 600 nodes.
Patiño wherein each graph comprises between 50 and 600 nodes. (Figure 2 “Segmentation results for three different images. From left to right: Original image with SLIC segmentation (k = 400), Merged superpixels, final segmentation binary mask, and ground-truth comparison. TP=light blue, TN=dark blue, FN: yellow and FP: red” Section 3.2 Superpixel segmentation “SLIC requires a parameter k indicating the desired number of superpixels in the resulting image…. In our experiments we empirically set k = 400.” Figure 1: “Fig. 1: SLIC segmentation for several k values. Left to right and top to bottom: k = 100,200,400,600” Section 3.3 “From this representation, a Region Adjacency Graph (RAG) is constructed. Each node is equipped with a list of properties derived from its RGB intensity values: Mean color, total color, and pixel count )
Accordingly, a person of ordinary skill in the art, at the time this invention was effectively filed would have found it obvious to modify the Thomas/Rahman methodology to include the representation of skin nodes in the range Patiño teaches. A person of ordinary skill in the art would do this to provide enough spatial resolution for representing skin feature boundaries while limiting unnecessary computational processing associated with excessive segmentation. This enables improved representation of skin feature boundaries and reduces computational cost. A person of ordinary skill in the art is are that Patiño is in the same realm of endeavor as the Thomas/Rahman system; identification of skin features within dermoscopic images in order to improve evaluation through time.
Claims 8 is rejected under 35 U.S.C. 103 as being unpatentable over Thomas et al (Thomas hereinafter US 9996923 B2) in view of Rahman et al (Rahman hereinafter US 20210118550 A1) in further view of Ng et al (Ng hereinafter “Determining the asymmetry of skin lesion with fuzzy borders”)
As per claim 8
Thomas teaches all claim limitations previously rejected in claim 7’s 102 rejection. See claim 7’s 102 rejection.
Rahman teaches wherein when a class is associated with a singularity after acquired images of the skin are supplied to a neural network configured to output a classification of said supplied images (Figure 3 Paragraph [0046] “Classification module 170 may be used to classify the images in multiple skin cancer categories”
Rahman nor Thomas teach one of the output classes from the list provided in claim 8
Ng teaches at least one class is comprised from the following list of classes: a class relating to the asymmetry of the geometry of the periphery of the singularity of a given dermoscopic image (Section 5.1 Backpropagation neural network: “Backpropagation neural networks have the ability to learn complicated multidimensional mappings and thus can be used to improve classification accuracy… The overall accuracy of classifying asymmetric lesions is about 80% (Table 4)” Ng specifically investigates how to identify asymmetry in the ABCD rule” and uses a “backpropagation neural network to improve discriminative powers of the measurement results.”)
Accordingly, a person of ordinary skill in the art, at the time this invention was effectively filed, would have found it obvious to modify the CNN in Rahman’s neural network based skin lesion classification system to classify a skin singularity according to lesion asymmetry as taught by Ng. Asymmetry is a known characteristic for distinguishing skin lesions and Ng expressly demonstrates neural network classification based on asymmetry measurements. This provides an additional clinically relevant characteristic for distinguishing skin lesions. This modification would have predictably improved the discrimination of skin lesions based on their geometric characteristics. This is particularly true when Rahman states in paragraph [0035] that “Transfer learning techniques can be used to extract features of dermoscopic images from a relatively small dataset using pretrained CNN models. Transfer learning increases the efficiency of the feature extraction process as it has been consistently proven to boost model accuracy with fewer data and reduce required training time. CNNs trained on large-scale datasets such as ImageNet have demonstrated to be excellent at the task of transfer learning. These networks learn a set of rich, discriminating features to recognize 1,000 separate object classes. Using a pretrained CNN as a feature extractor rather than training a CNN from scratch is attractive as it transfers learning (i.e., filters) from other domains where more training data is available and avoids a time consuming training process” This demonstrates the suitability of the disclosed neural network architecture for learning discriminative features for classification. Again, this yields improved discrimination and characterization of skin lesions using asymmetry as an additional classification characteristic.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANE WRENSFORD CODRINGTON whose telephone number is (571)272-8130. The examiner can normally be reached 8:00am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHANE WRENSFORD CODRINGTON/Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights