DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
35 U.S.C. 101 requires that a claimed invention must fall within one of the four eligible categories of invention (i.e. process, machine, manufacture, or composition of matter) and must not be directed to subject matter encompassing a judicially recognized exception as interpreted by the courts. MPEP 2106. The four eligible categories of invention include: (1) process which is an act, or a series of acts or steps, (2) machine which is an concrete thing, consisting of parts, or of certain devices and combination of devices, (3) manufacture which is an article produced from raw or prepared materials by giving to these materials new forms, qualities, properties, or combinations, whether by hand labor or by machinery, and (4) composition of matter which is all compositions of two or more substances and all composite articles, whether they be the results of chemical union, or of mechanical mixture, or whether they be gases, fluids, powders or solids. MPEP 2106(I).
Claims 1-4 are rejected under 35 U.S.C. 101 as not falling within one of the four statutory categories of invention because the claimed invention is directed to computer program per se. See MPEP 2106(I). A claim directed toward a non-transitory computer-readable medium having the program encoded thereon establishes a sufficient functional relationship between the program and a computer so as to remove it from the realm of “program per se”. MPEP 2111.05(III).
Claim 1 states: “A semi-automatic image segmentation system comprising a neural network configured to receive… interaction module configured to provide…”. Based on the specification p6, lines 10-20, p12, lines 5-10 this includes at least one purely software embodiment. Hence, adding the limitation of “stored on a non-transitory computer-readable medium” or similar language, or rewording, is necessary to resolve this issue.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 5, 6, and 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Forte et al. (IDS: “Getting to 99% Accuracy in Interactive Segmentation”).
Regarding claim 1, Forte et al. disclose a semi-automatic image segmentation system comprising a neural network configured to receive, at each iteration, input data comprising at least one image to be segmented (The loss for a particular image is defined, part 3, At each click iteration, we measure the proportion of correctly predicted pixels within the previous incorrect region, part 4.3, In Figure 14 and 15 we compare the predictions of our baseline and full algorithms for six clicks on images from the GrabCut and Berkeley datasets. Note that the baseline approach sometimes fails to recover from poor initial guesses, whereas our iteratively trained network is better at correcting with each click, part 4.4) and to provide output data comprising at least one segmentation mask of each image (Previous masks, Click map, predicted mask, Fig. 2, The segmentation masks are achieved through the use of an encoder-decoder network, The Interaction Stream encodes the previous mask together with the click maps, part 3.1), and a user interaction loop comprising a user interaction module configured to provide at least one user mask representative of an interaction of a real user and/or of a simulated user on each segmentation mask (The Interaction Stream encodes the previous mask together with the click maps, part 3.1
PNG
media_image1.png
604
708
media_image1.png
Greyscale
), said user interaction loop being configured to provide as input data of the neural network, wherein each segmentation mask provided by said neural network is coming from a previous iteration of the neural network, referred to as past segmentation mask, and the user mask is associated with each past segmentation mask (For each click, we first look at the previously predicted map, and segment the mislabelled pixels into connected regions. We place the click at the centre of the largest incorrect region, so as to maximise the Euclidean distance to both the region boundary and the sides of the image. For the first click, the prediction map is set to zero. The loss is then computed after the placement of each click and the weights are updated through back-propagation before the next click. By iterating through each click, we are essentially combining the loss for the entire range of clicks, hence effectively computing the area under the accuracy per clicks curve (as seen in Figures 3, 4 and 5), part 3.2) [click map interpreted as user mask].
Forte et al. do not use the language “user mask”. It would have been obvious at the time of filing to one of ordinary skill in the art as the user mask is a mask capturing user interaction, and the click map is used with the mask to predict a mask, the click map is equivalent to a user mask.
Regarding claim 2, Forte et al. disclose the segmentation system as claimed in claim 1. Forte et al. further indicate each user mask representative of a simulated user comprises a simulation of at least one interaction effected by the simulated operator on the segmentation mask associated with the user mask, the position of said interaction in the image being determined according to the difference between said output segmentation mask and ground truth (
PNG
media_image2.png
560
566
media_image2.png
Greyscale
, part 3.2).
Regarding claim 5, Forte et al. disclose a method of training an image segmentation system using at least one data set comprising images to be segmented, the system comprising: a neural network (Hi-Resolution Network Architecture, part 3.1, ResNet-50, Fig. 2, The Image Stream architecture is based on ResNet-50 with Group Normalisation and Weight Standardisation, part 3.1) configured to receive, at each iteration (By iterating through each click, we are essentially combining the loss for the entire range of clicks, part 3), input data comprising at least one image to be segmented (The loss for a particular image is defined, part 3, At each click iteration, we measure the proportion of correctly predicted pixels within the previous incorrect region, part 4.3, In Figure 14 and 15 we compare the predictions of our baseline and full algorithms for six clicks on images from the GrabCut and Berkeley datasets. Note that the baseline approach sometimes fails to recover from poor initial guesses, whereas our iteratively trained network is better at correcting with each click, part 4.4) and to provide output data comprising at least one segmentation mask of each image (Previous masks, Click map, predicted mask, Fig. 2, The segmentation masks are achieved through the use of an encoder-decoder network, The Interaction Stream encodes the previous mask together with the click maps, part 3.1), and a user interaction loop comprising a user interaction module configured to provide at least one user mask representative of an interaction of a real user and/or of a simulated user on each segmentation mask, said user interaction loop being configured to provide as input data of the neural network, wherein each segmentation mask provided by said neural network is coming from a previous iteration of the neural network, referred to as past segmentation mask, and the user mask is associated with each past segmentation mask (The Interaction Stream encodes the previous mask together with the click maps, part 3.1
PNG
media_image1.png
604
708
media_image1.png
Greyscale
) [click map interpreted as user map] [previous map = past segmentation map], wherein the method comprises for each image of the data set: at least two iterations (Il,12, In) in which the neural network processes the input data in order to provide output data, a first iteration (I1) in which the input data comprise at least one image to be segmented and a predefined initialization mask, and subsequent iterations in which the input data comprise each image to be segmented, each past segmentation mask obtained at the previous iteration, and a user mask representative of an interaction of a real user and/or of a simulated user (For each click, we first look at the previously predicted map, and segment the mislabelled pixels into connected regions. We place the click at the centre of the largest incorrect region, so as to maximise
the Euclidean distance to both the region boundary and the sides of the image. For the first click, the prediction map is set to zero. The loss is then computed after the placement of each click and the weights are updated through back-propagation before the next click. By iterating through each click, we are essentially combining the loss for the entire range of clicks, hence effectively computing the area under the accuracy per clicks curve (as seen in Figures 3, 4 and 5), part 3.2), a step of updating the weights of the neural network from the output data after each iteration or when a predetermined number of iterations have been effected by the neural network (The weights are initialised by pre-training on ImageNet for classification [32], part 3.1, The loss is then computed after the placement of each click and the weights are updated through back-propagation before the next click, part 3.2).
Forte et al. do not use the language “user map”. It would have been obvious at the time of filing to one of ordinary skill in the art as the user map is a map capturing user interaction, and the click map is used with the mask to predict a mask, the click map is equivalent to a user map.
Regarding claim 6, Forte et al. disclose the training method as claimed in claim 5. Forte et al. further indicate the predetermined number of iterations is determined for each image to be segmented and for each data set independently of the predetermined number of iterations determined for the other images to be segmented and the other data sets (A fundamental deviation from previous works is that we train our network, image by image, click by click, By iterating through each click, we are essentially combining the loss for the entire range of clicks, hence effectively computing the area under the accuracy per clicks curve (as seen in Figures 3, 4 and 5). This has the advantage that our training loss matches the final metric used for evaluation, part 3.2) [as each image computed independently and loss evaluated for each image this implies iterations determined independently of other images].
Regarding claim 7, Forte et al. disclose a semi-automatic image segmentation method comprising, at each iteration, a step of processing input data comprising each image (The loss for a particular image is defined, part 3, At each click iteration, we measure the proportion of correctly predicted pixels within the previous incorrect region, part 4.3, In Figure 14 and 15 we compare the predictions of our baseline and full algorithms for six clicks on images from the GrabCut and Berkeley datasets. Note that the baseline approach sometimes fails to recover from poor initial guesses, whereas our iteratively trained network is better at correcting with each click, part 4.4) to be segmented by a neural network (Hi-Resolution Network Architecture, part 3.1, ResNet-50, Fig. 2, The Image Stream architecture is based on ResNet-50 with Group Normalisation and Weight Standardisation, part 3.1) in order to provide output data comprising at least one segmentation mask of each image (Previous masks, Click map, predicted mask, Fig. 2, The segmentation masks are achieved through the use of an encoder-decoder network, The Interaction Stream encodes the previous mask together with the click maps, part 3.1), wherein each iteration further comprises: a step of receiving a user mask representative of an interaction of a real user on each segmentation mask, a step of transmitting, as input data of the neural network at the following iteration, each segmentation mask provided by said neural network, and the user mask (The Interaction Stream encodes the previous mask together with the click maps, part 3.1
PNG
media_image1.png
604
708
media_image1.png
Greyscale
For each click, we first look at the previously predicted map, and segment the mislabelled pixels into connected regions. We place the click at the centre of the largest incorrect region, so as to maximise the Euclidean distance to both the region boundary and the sides of the image. For the first click, the prediction map is set to zero. The loss is then computed after the placement of each click and the weights are updated through back-propagation before the next click. By iterating through each click, we are essentially combining the loss for the entire range of clicks, hence effectively computing the area under the accuracy per clicks curve (as seen in Figures 3, 4 and 5), part 3.2) [click map interpreted as user mask].
Forte et al. do not use the language “user mask”. It would have been obvious at the time of filing to one of ordinary skill in the art as the user mask is a mask capturing user interaction, and the click map is used with the mask to predict a mask, the click map is equivalent to a user mask.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Forte et al. (IDS: “Getting to 99% Accuracy in Interactive Segmentation”) in view of Birbeck et al. (“An Interactive Graph Cut Method for Brain Tumor Segmentation”).
Regarding claim 3, Forte et al. disclose the segmentation system as claimed in claim 1. Forte et al. do not disclose a graph cut module configured to apply a graph cut to the output data of the neural network in order to obtain each segmentation mask.
Birbeck et al. teach a graph cut module configured to apply a graph cut to the output data of the neural network in order to obtain each segmentation mask (
PNG
media_image3.png
472
780
media_image3.png
Greyscale
PNG
media_image4.png
202
392
media_image4.png
Greyscale
PNG
media_image5.png
604
386
media_image5.png
Greyscale
).
Forte et al. and Birbeck et al. are in the same art of user interaction and segmentation masks (Forte et al., abstract; Birbeck et al., abstract). The combination of Birbeck et al. with Forte et al. will enable using graph cuts. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the graph cuts of Birbeck et al. with the invention of Forte et al. as this was known at the time of filing, the combination would have predictable results, and as Birbeck et al. indicate “The energy is efficiently minimized on-line using graph cut. Experiments with radiation oncologists testing the semi-automatic tool vs. a manual tool show that the proposed system improves both segmentation time and repeatability” (abstract) providing an efficiency benefit to combining inventions.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Forte et al. (IDS: “Getting to 99% Accuracy in Interactive Segmentation”) in view of Blake et al. (US 20060285747 A1).
Regarding claim 4, Forte et al. disclose the segmentation system as claimed in claim 1. Forte et al. partly disclose the user interaction loop comprises an internal sequential memory in which each past segmentation mask and each user mask obtained at each iteration are stored sequentially (The Interaction Stream encodes the previous mask together with the click maps, part 3.1
PNG
media_image1.png
604
708
media_image1.png
Greyscale
For each click, we first look at the previously predicted map, and segment the mislabelled pixels into connected regions. We place the click at the centre of the largest incorrect region, so as to maximise the Euclidean distance to both the region boundary and the sides of the image. For the first click, the prediction map is set to zero. The loss is then computed after the placement of each click and the weights are updated through back-propagation before the next click. By iterating through each click, we are essentially combining the loss for the entire range of clicks, hence effectively computing the area under the accuracy per clicks curve (as seen in Figures 3, 4 and 5), we propose to introduce the clicks sequentially, starting from a single click, and adopting the same sequential scheme used to evaluate interactive segmentation algorithms, part 3.2) but another reference is assed to teach the term “sequential memory”.
Blake et al. teach a user interaction loop comprises an internal sequential memory in which each past segmentation mask and each user mask obtained at each iteration are stored sequentially (A data store may include one or more of a relational database, object-oriented database, unstructured database, an in-memory database, sequential memory, or other data store. A storage array is a form of a data store and may be constructed using a flat file system such as ASCII text, a binary file, data transmitted across a communication network, or any other file system, [0036], interactively obtained segmentation masks 320, 340 respectively, [0070], For example, a graph cut algorithm may be used to solve the Markov Random Field to produce the accurate segmentation masks, [0082]).
Forte et al. and Blake et al. are in the same art of user interaction and segmentation masks (Forte et al., abstract; Blake et al., [0070]). The combination of Blake et al. with Forte et al. will enable using sequential memory. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the sequential memory of Blake et al. with the invention of Forte et al. as this was known at the time of filing, the combination would have predictable results, as this is one of a limited number of types of memory, and as Blake et al. indicate “Automatic separation of layers from color/contrast or from motion alone may include errors. To reduce segmentation errors, color, motion, and optionally contrast information may be probabilistically fused to infer foreground and/or background layers accurately and efficiently. In this manner, pixel velocities are not needed. Therefore, a number of issues related to optical flow estimation are removed. Instead, a likelihood of motion vs. non-motion may be automatically learned from training data and then fused with a contrast-sensitive color model. Segmentation may then be solved efficiently by an optimization algorithm such as a graph cut” ([0004]) indicating an efficiency benefit to combining inventions.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE M ENTEZARI HAUSMANN whose telephone number is (571)270-5084. The examiner can normally be reached 10-7 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent M Rudolph can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE M ENTEZARI HAUSMANN/Primary Examiner, Art Unit 2671
Prosecution Insights