DETAILED ACTIONS
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/18/2025 has been entered.
Response to Arguments
Applicant’s arguments, see pages 9-10, filed 09/20/2025, with respect to the rejection of claim 1 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new grounds of rejection is made in view of normalizing a 3D image of a subject to a reference size and shape wherein the reference size and shape are determined based on at least one reference image.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 9-10, and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Gladilin et al., "Shape normalization of 3D cell nuclei using elastic spherical mapping" (2008), hereinafter referred to as Gladilin, in view of Aragaki et al., (US 2019/0095679 A1), hereinafter referred to as Aragaki.
Claim 1
Gladilin discloses an image processing system for standardizing the size and shape of organisms (Gladilin, Title, Fig. 4), the system comprising:
a camera configured to output images of a subject (Gladilin, page 106, “Multichannel 3D confocal laser scanning microscopy (CLSM) images of human fibroblast nuclei are used for analysing the geometrical disposition of gene loci within the cell nucleus, as shown in Fig. 1.”);
receive a set of three-dimensional images of the subject from the camera (Gladilin, page 106, “Multichannel 3D confocal laser scanning microscopy (CLSM) images of human fibroblast nuclei are used for analysing the geometrical disposition of gene loci within the cell nucleus, as shown in Fig. 1.”, Fig. 4, original image); and
process the set of three-dimensional images with a computational model (Gladilin, Fig. 4, the surface template alignment is a computational model) to normalize them to a reference size and shape to generate a set of normalized images (Gladilin, page 111, “The approach described above was applied for normalization of four-channel 3D CLSM images of human fibroblast nuclei including overall nuclear shape and three genomic regions. These three BAC regions correspond to (1) one region of increased gene expression (ridge), (2) one gene-sparse region (named antiridge) and (3) centromere of the human chromosome 1 (Goetze et al. 2007), as shown in Fig. 7”, the sizes of these genomic regions are also normalized as shown in Table 1, page 113, “Table 1 summarises the results of the calculation of relative radial distances (RRD) of three BAC regions labelled along the human chromosome 1 in six interphase nuclei [two chromosomes 1 (A,B) per each nucleus] before (top) and after (bottom) their shape normalization via elastic spherical mapping, respectively”) wherein the reference size and shape are determined based on at least one reference image (Gladilin, Fig. 4, “ellipsoidal surface template mesh corresponding to the eigenellipsoid of the binariz’ed image (top, middle)”, the ellipsoidal surface template is analogous to the reference image in which the original image is being aligned to generate a normalized shape and size, page 113, “We presented a novel approach for unsupervised normalization of multichannel 3D CLSM images of cell nuclei by means of an inverse spherical mapping. A FE template is used for affine and elastic registration of a spherical mesh with segmented 3D nuclear images.”, “Such spherical mapping provides a consistent shape normalization for further quantitative analysis of intranuclear disposition of targeted structures in terms of canonical spherical coordinates, for example, nondimensional relative radial distances”).
Glaldilin does not explicitly disclose a memory in communication with the camera containing machine readable medium comprising machine executable code having stored thereon and a control system comprising one or more processors coupled to the memory, the control system configured to execute the machine executable to cause the control system receive a set of three-dimensional images of the subject from the camera and process the set of three-dimensional images with a computational model to normalize them.
However, Aragaki teaches a memory (Aragaki, [0034], “As shown in FIG. 1, the image processing device 5 includes a normalizing unit 7, a reducing unit 9, a first smoothing processor (smoothing unit) 11, a seed creating unit (binarizing unit) 13, a grayscale correcting unit 15, a second smoothing processor (smoothing unit) 17, a schematic-shape-region creating unit 19, a region-expansion reference-image creating unit 21, a region expanding unit (region segmenting unit) 23, an enlarging unit 25, a filter (measuring unit, filtering unit) 27, and an output unit 29. These units are connected to and operationally controlled by a system controller (not shown). Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”) in communication with the camera containing machine readable medium comprising machine executable code having stored thereon and a control system comprising one or more processors (Aragaki, [0034], “Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”) coupled to the memory (Aragaki, [0034], “Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”), the control system configured to execute the machine executable to cause the control system (Aragaki, [0034], “Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.” to receive a set of three-dimensional images of the subject from the camera and process the set of three-dimensional images with a computational model to normalize them ([0035], “The normalizing unit 7 reads an original image signal (three-dimensional z-stack image data), which is formed by capturing an image of a cell cluster (spheroid S), sent from the image acquisition device 3 and normalizes each pixel value within a predetermined grayscale range, such as a 12 bit (0-4095 grayscale) range, so as to generate a normalized image. The normalization process is performed in accordance with, for example, a histogram spreading process.”).
Gladilin and Aragaki are both considered to be analogous to the claimed invention because they are in the same field of image normalization. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system as taught by Gladilin to incorporate the teachings of Aragaki a memory in communication with the camera containing machine readable medium comprising machine executable code having stored thereon and a control system comprising one or more processors coupled to the memory, the control system configured to execute the machine executable to cause the control system receive a set of three-dimensional images of the subject from the camera and process the set of three-dimensional images with a computational model to normalize them. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been to make the system more robust.
Claim 2
The combination of Gladilin in view of Aragaki discloses the system of claim 1 (Gladilin, Title, Fig. 4), wherein the camera is a three-dimensional camera and the set of images of the subject are depth images (Gladilin, page 106, “Multichannel 3D confocal laser scanning microscopy (CLSM) images of human fibroblast nuclei are used for analysing the geometrical disposition of gene loci within the cell nucleus, as shown in Fig. 1.”).
Claims 9 and 10 are rejected for similar reasons as those described in claims 1 and 2. The additional elements in Claims 9 and 10 (Gladilin in view of Aragaki) discloses includes: a method for standardizing the size and shape of organisms (Gladilin, Fig. 4). The proposed combination as well as the motivation for combining the Gladilin and Aragaki references presented in the rejection of Claim 1, apply to Claims 9 and 10 and are incorporated herein by reference. Thus, the method recited in Claims 9 and 10 is met by Gladilin and Aragaki.
Claims 17 and 18 are rejected for similar reasons as those described in claims 1 and 2. The additional elements in Claims 17 and 18 (Gladilin in view of Aragaki) discloses includes: a non-transitory machine readable medium (Aragaki, [0034], “Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”) having stored thereon instructions for performing a method (Gladilin, Fig. 4) comprising machine executable code which when executed by at least one machine (Aragaki, [0034], “As shown in FIG. 1, the image processing device 5 includes a normalizing unit 7, a reducing unit 9, a first smoothing processor (smoothing unit) 11, a seed creating unit (binarizing unit) 13, a grayscale correcting unit 15, a second smoothing processor (smoothing unit) 17, a schematic-shape-region creating unit 19, a region-expansion reference-image creating unit 21, a region expanding unit (region segmenting unit) 23, an enlarging unit 25, a filter (measuring unit, filtering unit) 27, and an output unit 29. These units are connected to and operationally controlled by a system controller (not shown). Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”). The proposed combination as well as the motivation for combining the Gladilin and Aragaki references presented in the rejection of Claim 1, apply to Claims 17 and 18 and are incorporated herein by reference. Thus, the method recited in Claims 17 and 18 is met by Gladilin and Aragaki.
Claims 3-4, 6, 11-12, 14, 19-20, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Gladilin in view of Aragaki in further view of Sun et al., “Human Pose Estimation using Global and Local Normalization” (2017), hereinafter referred to as Sun.
Claim 3
The combination of Gladilin in view of Aragaki discloses the system of claim 1 (Gladilin, Title, Fig. 4)
The combination of Gladilin in view of Aragaki does not explicitly disclose wherein the computational model is a deep neural network.
However, Sun teaches wherein the computational model is a deep neural network (Sun, Fig. 2, “Proposed framework with global and local normalization. Joint detection with fully convolutional network (FCN) provides initial estimation of joints in terms of score maps. In the global refinement stage, a body (global) normalization module rotates the score maps to have upright position for the body, followed by a refinement module. In the local refinement stage, limb (local) normalization modules rotates the score maps to have vertical downward position for limbs, followed by refinements”).
Gladilin, Aragaki, and Sun are all considered to be analogous to the claimed invention because they are in the same field of normalization. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system as taught by Gladilin in view of Aragaki to incorporate the teachings of Sun wherein the computational model is a deep neural network. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have because it results in easier learning of convolutional spatial models and more accurate pose estimation (Sun, Abstract).
Claim 4
Gladilin discloses an image processing system for standardizing the size and shape of organisms (Gladilin, Title, Fig. 4), the system comprising:
a camera configured to output images of a subject (Gladilin, page 106, “Multichannel 3D confocal laser scanning microscopy (CLSM) images of human fibroblast nuclei are used for analysing the geometrical disposition of gene loci within the cell nucleus, as shown in Fig. 1.”);
receive a set of three-dimensional images of the subject from the camera (Gladilin, page 106, “Multichannel 3D confocal laser scanning microscopy (CLSM) images of human fibroblast nuclei are used for analysing the geometrical disposition of gene loci within the cell nucleus, as shown in Fig. 1.”, Fig. 4, original image); and
process the set of three-dimensional images with a computational model (Gladilin, Fig. 4, the surface template alignment is a computational model) to normalize them to a reference size and shape to generate a set of normalized images (Gladilin, page 111, “The approach described above was applied for normalization of four-channel 3D CLSM images of human fibroblast nuclei including overall nuclear shape and three genomic regions. These three BAC regions correspond to (1) one region of increased gene expression (ridge), (2) one gene-sparse region (named antiridge) and (3) centromere of the human chromosome 1 (Goetze et al. 2007), as shown in Fig. 7”, the sizes of these genomic regions are also normalized as shown in Table 1, page 113, “Table 1 summarises the results of the calculation of relative radial distances (RRD) of three BAC regions labelled along the human chromosome 1 in six interphase nuclei [two chromosomes 1 (A,B) per each nucleus] before (top) and after (bottom) their shape normalization via elastic spherical mapping, respectively”) wherein the reference size and shape are determined based on at least one reference image (Gladilin, Fig. 4, “ellipsoidal surface template mesh corresponding to the eigenellipsoid of the binariz’ed image (top, middle)”, the ellipsoidal surface template is analogous to the reference image in which the original image is being aligned to generate a normalized shape and size, page 113, “We presented a novel approach for unsupervised normalization of multichannel 3D CLSM images of cell nuclei by means of an inverse spherical mapping. A FE template is used for affine and elastic registration of a spherical mesh with segmented 3D nuclear images.”, “Such spherical mapping provides a consistent shape normalization for further quantitative analysis of intranuclear disposition of targeted structures in terms of canonical spherical coordinates, for example, nondimensional relative radial distances”).
Glaldilin does not explicitly disclose a memory in communication with the camera containing machine readable medium comprising machine executable code having stored thereon and a control system comprising one or more processors coupled to the memory, the control system configured to execute the machine executable to cause the control system receive a set of three-dimensional images of the subject from the camera and process the set of three-dimensional images with a computational model to normalize them.
However, Aragaki teaches a memory (Aragaki, [0034], “As shown in FIG. 1, the image processing device 5 includes a normalizing unit 7, a reducing unit 9, a first smoothing processor (smoothing unit) 11, a seed creating unit (binarizing unit) 13, a grayscale correcting unit 15, a second smoothing processor (smoothing unit) 17, a schematic-shape-region creating unit 19, a region-expansion reference-image creating unit 21, a region expanding unit (region segmenting unit) 23, an enlarging unit 25, a filter (measuring unit, filtering unit) 27, and an output unit 29. These units are connected to and operationally controlled by a system controller (not shown). Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”) in communication with the camera containing machine readable medium comprising machine executable code having stored thereon and a control system comprising one or more processors (Aragaki, [0034], “Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”) coupled to the memory (Aragaki, [0034], “Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”), the control system configured to execute the machine executable to cause the control system (Aragaki, [0034], “Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.” to receive a set of three-dimensional images of the subject from the camera and process the set of three-dimensional images with a computational model to normalize them ([0035], “The normalizing unit 7 reads an original image signal (three-dimensional z-stack image data), which is formed by capturing an image of a cell cluster (spheroid S), sent from the image acquisition device 3 and normalizes each pixel value within a predetermined grayscale range, such as a 12 bit (0-4095 grayscale) range, so as to generate a normalized image. The normalization process is performed in accordance with, for example, a histogram spreading process.”).
Gladilin and Aragaki are both considered to be analogous to the claimed invention because they are in the same field of image normalization. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system as taught by Gladilin to incorporate the teachings of Aragaki a memory in communication with the camera containing machine readable medium comprising machine executable code having stored thereon and a control system comprising one or more processors coupled to the memory, the control system configured to execute the machine executable to cause the control system receive a set of three-dimensional images of the subject from the camera and process the set of three-dimensional images with a computational model to normalize them. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been to make the system more robust.
The combination of Gladilin in view of Aragaki does not explicitly disclose the computational model was trained by first manipulating a size and shape of a training subject in a set of training images to output a set of manipulated training images and training the computational model to process the set of manipulated training images to match an original image from the set of training images to output a restored set of images in which the training subject has been restored to its original size and shape from the set of training images.
However, Sun teaches the computational model (Sun, Fig. 2) was trained by first manipulating a size and shape of a training subject in a set of training images to output a set of manipulated training images (Sun, page 5602, “Specifically, for the alternative two solutions, we process the training data with extra data augmentation, to make the number of training data for each type similar to ours”) and training the computational model to process the set of manipulated training images to match an original image from the set of training images to output a restored set of images in which the training subject has been restored to its original size and shape from the set of training images (Sun, page 5603, “End-to-end training is supported with the error back propagation along the transform path (as denoted by the green line). For the joint position determination module, a Gaussian blur is performed on the mapped score maps, with the mapping corresponding to the Sigmoid like operation or no operation, depending the loss design of the joint detection network. Then, the position corresponding to the maximal value in each processed score map is estimated as the position of that joint. The network calculates the rotation center c and the rotation angle θ based on the estimated positions of joints. All the operations are incorporated into the network as layers.”, Table 1, “Comparing our global normalization-based solution with type supervision and the multi-branch solution on LSP dataset with the OC annotation (@PCK 0.2) trained on the LSP dataset.”, the original size is derived from the training dataset).
Gladilin, Aragaki, and Sun are all considered to be analogous to the claimed invention because they are in the same field of normalization. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the image signal processor as taught by Gladilin in view of Aragaki to incorporate the teachings of Sun wherein the computational model was trained by first manipulating a size and shape of a training subject in a set of training images to output a set of manipulated training images and training the computational model to process the set of manipulated training images to match an original image from the set of training images to output a restored set of images in which the training subject has been restored to its original size and shape from the set of training images. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have because it results in easier learning of convolutional spatial models and more accurate pose estimation (Sun, Abstract).
Claim 6
The combination of Gladilin in view of Aragaki in further view of Sun discloses the system of claim 4 (Gladilin, Title, Fig. 4), wherein first manipulating the size and shape comprises altering the position, rotation, length, width, height, and/or aspect ratio of the training subject (Sun, page 5604, “Data Augmentation. For the LSP dataset, we augment the training data by performing random scaling with a scaling factor between 0.80 and 1.25, horizontal flipping, and rotating the data across 360 degrees, in consideration of its unbalanced distribution of pose orientations. All input images are resized to 340 × 340 pixels. For the FLIC and the MPII dataset, we randomly rotate the data across +/- 30 degrees and resize images into 256 × 256 pixels”). The proposed combination as well as the motivation for combining the Gladilin, Aragaki, and Sun references presented in the rejection of Claim 3, apply to Claim 6 and are incorporated herein by reference. Thus, the system recited in Claim 6 is met by Gladilin, Aragaki, and Sun.
Claims 11-12 and 14 are rejected for similar reasons as those described in claims 3-4 and 6. The additional elements in Claims 11-12 and 14 (Gladilin, Aragaki, and Sun) discloses includes: a method for standardizing the size and shape of organisms (Gladilin, Title, Fig. 4). The proposed combination as well as the motivation for combining the Gladilin, Aragaki, and Sun references presented in the rejection of Claims 3-4 and 6, apply to Claim 11-12 and 14 and are incorporated herein by reference. Thus, the method recited in Claim 11-12 and 14 is met by Gladilin, Aragaki, and Sun.
Claims 19-20 and 22 are rejected for similar reasons as those described in claims 3-4 and 6. The additional elements in Claims 19-20 and 22 (Gladilin, Aragaki, and Sun) discloses includes: a non-transitory machine readable medium (Aragaki, [0034], “Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”) having stored thereon instructions for performing a method (Gladilin, Fig. 4) comprising machine executable code which when executed by at least one machine (Aragaki, [0034], “As shown in FIG. 1, the image processing device 5 includes a normalizing unit 7, a reducing unit 9, a first smoothing processor (smoothing unit) 11, a seed creating unit (binarizing unit) 13, a grayscale correcting unit 15, a second smoothing processor (smoothing unit) 17, a schematic-shape-region creating unit 19, a region-expansion reference-image creating unit 21, a region expanding unit (region segmenting unit) 23, an enlarging unit 25, a filter (measuring unit, filtering unit) 27, and an output unit 29. These units are connected to and operationally controlled by a system controller (not shown). Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”). The proposed combination as well as the motivation for combining the Gladilin, Aragaki, and Sun references presented in the rejection of Claims 3-4 and 6, apply to Claim 19-20 and 22 and are incorporated herein by reference. Thus, the medium recited in Claim 19-20 and 22 is met by Gladilin, Aragaki, and Sun.
Claims 5, 13, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Gladilin in view of Aragaki in further view of Sun in view of Odry et al., (US 2017/0371017 A1), hereinafter referred to as Odry.
Claim 5
The combination of Gladilin in view of Aragaki in further view of Sun discloses the system of claim 3 (Gladilin, Title, Fig. 4).
The combination of Gladilin in view of Aragaki in further view of Sun does not explicitly disclose wherein the deep neural network comprises a denoising convolutional autoencoder and a U-NET.
However, Odry teaches wherein the deep neural network comprises a denoising convolutional autoencoder and a U-NET (Odry, [0034], “an autoencoder such as a VAE (though not limited to a VAE) may be used in a process and a system to learn a transformation φ(X,θ) that takes MR images as an input and match those input images to the normalized database of MR images (i.e., the target)”, [0011], “FIG. 6 is a system diagram of a task network including some aspects of a normalization network, according to some embodiments”).
Gladilin, Aragaki, Sun, and Odry are all considered to be analogous to the claimed invention because they are in the same field of normalizing images. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system as taught by Gladilin, Aragaki, and Sun to incorporate the teachings of Odry wherein the deep neural network comprises a denoising convolutional autoencoder and a U-NET. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have because MR images with normalized intensities of a brain can be accurately viewed/interpreted (Odry, [0027]).
Claim 13 is rejected for similar reasons as those described in claim 5. The additional elements in Claim 13 (Gladilin, Aragaki, Sun and Odry) discloses includes: a method for standardizing the size and shape of organisms (Gladilin, Title, Fig. 4). The proposed combination as well as the motivation for combining the Gladilin, Aragaki, Sun, and Odry references presented in the rejection of Claim 5, apply to Claim 13 and are incorporated herein by reference. Thus, the method recited in Claim 13 is met by Gladilin, Aragaki, Sun, and Odry.
Claim 21 is rejected for similar reasons as those described in claim 5. The additional elements in Claim 21 (Gladilin, Aragaki, Sun and Odry) discloses includes: a non-transitory machine readable medium (Aragaki, [0034], “Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”) having stored thereon instructions for performing a method (Gladilin, Fig. 4) comprising machine executable code which when executed by at least one machine (Aragaki, [0034], “As shown in FIG. 1, the image processing device 5 includes a normalizing unit 7, a reducing unit 9, a first smoothing processor (smoothing unit) 11, a seed creating unit (binarizing unit) 13, a grayscale correcting unit 15, a second smoothing processor (smoothing unit) 17, a schematic-shape-region creating unit 19, a region-expansion reference-image creating unit 21, a region expanding unit (region segmenting unit) 23, an enlarging unit 25, a filter (measuring unit, filtering unit) 27, and an output unit 29. These units are connected to and operationally controlled by a system controller (not shown). Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”). The proposed combination as well as the motivation for combining the Gladilin, Aragaki, and Sun references presented in the rejection of Claim 5, apply to Claim 21 and are incorporated herein by reference. Thus, the medium recited in Claim 21 is met by Gladilin, Aragaki, and Sun.
Claims 7, 15, and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Gladilin in view of Aragaki in further view of Rad et al., (US 2017/0262962 A1), hereinafter referred to as Rad.
Claim 7
The combination of Gladilin in view of Aragaki discloses the system of claim 1 (Gladilin, Title, Fig. 4).
The combination of Gladilin in view of Aragaki does not explicitly disclose wherein the control system is further configured to: process the set of normalized images using another computational model to partition images of the set of normalized images into at least one set of images that represent modules and at least one set of images that represent transitions between the modules; and storing, in a memory, the at least one set of images that represent modules referenced to a data identifier that represents a type of animal behavior.
However, Rad teaches wherein the control system (Rad, [0067], “FIG. 1 is a block diagram illustrating one example of an electronic device 102 in which systems and methods for normalizing an image may be implemented”) is further configured to: process the set of normalized images using another computational model to partition images of the set of normalized images into at least one set of images that represent modules and at least one set of images that represent transitions between the modules (Rad, Fig. 3, the normalized images are used as an input to a detector neural network); and storing, in a memory (Rad, [0069], “In some configurations, the electronic device 102 may include a processor 112, a memory 122, a display 124, one or more image sensors 104, one or more optical systems 106, and/or one or more communication interfaces 108”), the at least one set of images that represent modules referenced to a data identifier that represents a type of animal behavior (Rad, [0100], “the electronic device 102 may detect an object in an image, track an object in a series of images (e.g., video), modify an image, recognize one or more objects in an image, automatically focus one or more optical systems (e.g., lenses), automatically zoom in on a detected object, drive autonomously (e.g., read signs, detect pedestrians, detect obstacles for avoidance, detect other vehicles, control a drive train, control braking, steering, etc.), assist a driver (e.g., apply emergency braking, perform an emergency maneuver, alert a driver of a sign, alert a driver of an obstacle, etc.), perform navigation (e.g., plan a path for a robot or drone, assemble an object, disassemble an object, etc.), etc., based on one or more normalized images and/or normalized image windows”, [0114], “Additionally or alternatively, one or more normalized windows may be provided to a computer vision application (for detecting an object, autofocus, auto-zoom, object tracking, object identification, object recognition, etc.)”)
Gladilin, Aragaki and Rad are all considered to be analogous to the claimed invention because they are in the same field of normalizing images. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as taught by Gladilin and Aragaki to incorporate the teachings of Rad wherein the control system is further configured to: process the set of normalized images using another computational model to partition images of the set of normalized images into at least one set of images that represent modules and at least one set of images that represent transitions between the modules; and storing, in a memory, the at least one set of images that represent modules referenced to a data identifier that represents a type of animal behavior. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have because by normalizing each image window depending on its characteristic, it results in improving the accuracy of the detector (Rad, [0065]).
Claim 15 is rejected for similar reasons as those described in claim 7. The additional elements in Claim 15 (Gladilin, Aragaki and Rad) discloses includes: a method for standardizing the size and shape of organisms (Gladilin, Title, Fig. 4). The proposed combination as well as the motivation for combining the Gladilin, Aragaki and Rad references presented in the rejection of Claim 7, apply to Claim 15 and are incorporated herein by reference. Thus, the method recited in Claim 15 is met by Gladilin, Aragaki and Rad.
Claim 23 is rejected for similar reasons as those described in claim 7. The additional elements in Claim 23 (Gladilin, Aragaki and Rad) discloses includes: a non-transitory machine readable medium (Aragaki, [0034], “Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”) having stored thereon instructions for performing a method (Gladilin, Fig. 4) comprising machine executable code which when executed by at least one machine (Aragaki, [0034], “As shown in FIG. 1, the image processing device 5 includes a normalizing unit 7, a reducing unit 9, a first smoothing processor (smoothing unit) 11, a seed creating unit (binarizing unit) 13, a grayscale correcting unit 15, a second smoothing processor (smoothing unit) 17, a schematic-shape-region creating unit 19, a region-expansion reference-image creating unit 21, a region expanding unit (region segmenting unit) 23, an enlarging unit 25, a filter (measuring unit, filtering unit) 27, and an output unit 29. These units are connected to and operationally controlled by a system controller (not shown). Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”). The proposed combination as well as the motivation for combining the Gladilin, Aragaki and Rad references presented in the rejection of Claim 7, apply to Claim 23 and are incorporated herein by reference. Thus, the medium recited in Claim 23 is met by Gladilin, Aragaki and Rad.
Claims 8, 16, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Gladilin in view of Aragaki in view of Rad in further view of Datta et al., (WO 2017/161167 A1), hereinafter referred to as Datta.
Claim 8
The combination of Gladilin in view of Aragaki in further view of Rad discloses the system of claim 7 (Gladilin, Title, Fig. 4).
The combination of Gladilin in view of Aragaki in further view of Rad does not explicitly disclose wherein the control system is further configured to: pre-process, using the control system, the set of normalized images to isolate the subject from a background; identify, using the control system, an orientation of a feature of the subject in the set of normalized images with respect to a coordinate system common to each image of the set of normalized images; modify, using the control system, the orientation of the subject in at least a subset of the set of normalized so that the feature is oriented in a same direction with respect to the coordinate system to output a set of aligned images; and process, using the control system, the set of aligned images using a principal component analysis to output pose dynamics data for each images of the set of aligned images, wherein the pose dynamics data represents a pose of the subject for each aligned image through principal component space.
However, Datta teaches wherein the control system (Datta, [0085], “the data may be processed by an associated computer 113”) is further configured to: pre-process, using the control system, the set of normalized images to isolate the subject from a background (Datta, [0087], “various pre-processing may take place to isolate the animal in the video data and orient the images of the animal along a common axis for further processing”); identify, using the control system (Datta, [0085], “the data may be processed by an associated computer 113”), an orientation of a feature of the subject in the set of normalized images with respect to a coordinate system common to each image of the set of normalized images (Datta, [0087], “In some embodiments, the orientation of the head may be utilized to orient the images in a common direction. In other embodiments, an inferred direction of the spine may be incorporated”, [0094], “The centroid of the animal (e.g. mouse) may then be identified 270 as the center-of-mass of the preprocessed image or by other suitable methods; an ellipse may then be fit to its contour 285 to detect its overall orientation. In order to properly orient the mouse 280, various machine learning algorithms may be trained (e.g. a random forest classifier) on a set of manually-oriented extracted mouse images. Given an image, the orientation algorithm then returns an output indicating whether the mouse's head is oriented correctly or not.”) ; modify, using the control system (Datta, [0085], “the data may be processed by an associated computer 113”), the orientation of the subject in at least a subset of the set of normalized so that the feature is oriented in a same direction with respect to the coordinate system to output a set of aligned images (Datta, [0095], “Once the position is identified, additional information may be extracted 275 from the video data including the centroid, head and tail positions of the animal, orientation, length, width, height, and each of their first derivatives with respect to time. Characterization of the animal's pose dynamics required correction of perspective distortion in the X and Y axes. This distortion may be corrected by first generating a tuple of (x,y,z) coordinates for each pixel in real-world coordinates, and then resampling those coordinates to fall on an even grid in the (x,y) plane using Delaunay tri angulation”); and process, using the control system, the set of aligned images using a principal component analysis to output pose dynamics data for each images of the set of aligned images (Datta [0020], “To identify similar modules, mouse behavioral data may first be subject dimensionality reduction using, for example (1) principal component analysis, and (2) neural networks such as multi-layer perceptrons. For instance, using principal components analysis (PCA), the first two principal components may be plotted. Each block in the pose dynamics data corresponds to a continuous trajectory through PCA space; for example, an individual block associated with the mouse's spine being elevated corresponded to a specific sweep through PCA space. Scanning the behavioral data for matching motifs using a template matching method identified several additional examples of this sweep in different animals, suggesting that each of these PCA trajectories may represent individual instances in which a stereotyped behavioral module was reused.”), wherein the pose dynamics data represents a pose of the subject for each aligned image through principal component space (Datta, [0020], “Each block in the pose dynamics data corresponds to a continuous trajectory through PCA space; for example, an individual block associated with the mouse's spine being elevated corresponded to a specific sweep through PCA space. Scanning the behavioral data for matching motifs using a template matching method identified several additional examples of this sweep in different animals, suggesting that each of these PCA trajectories may represent individual instances in which a stereotyped behavioral module was reused.”).
Gladilin, Aragaki, Rad and Datta are all considered to be analogous to the claimed invention because they are in the same field of image processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as taught by Gladilin, Aragaki and Rad to incorporate the teachings of Datta wherein the control system is further configured to: pre-process, using the control system, the set of normalized images to isolate the subject from a background; identify, using the control system, an orientation of a feature of the subject in the set of normalized images with respect to a coordinate system common to each image of the set of normalized images; modify, using the control system, the orientation of the subject in at least a subset of the set of normalized so that the feature is oriented in a same direction with respect to the coordinate system to output a set of aligned images; and process, using the control system, the set of aligned images using a principal component analysis to output pose dynamics data for each images of the set of aligned images, wherein the pose dynamics data represents a pose of the subject for each aligned image through principal component space. The motivation for the proposed modification would have because it offers insight into the nature of behavioral change in the subject, and enables objective discovery of subtle alterations in patterned action (Datta, Abstract).
Claim 16 is rejected for similar reasons as those described in claim 8. The additional elements in Claim 16 (Gladilin, Aragaki, Rad, and Datta) discloses includes: a method for standardizing the size and shape of organisms (Gladilin, Title, Fig. 4). The proposed combination as well as the motivation for combining the Gladilin, Aragaki, Rad, and Datta references presented in the rejection of Claim 8, apply to Claim 16 and are incorporated herein by reference. Thus, the method recited in Claim 16 is met by Gladilin, Aragaki, Rad, and Datta.
Claim 24 is rejected for similar reasons as those described in claim 8. The additional elements in Claim 24 (Gladilin, Aragaki, Rad, and Datta) discloses includes: a non-transitory machine readable medium (Aragaki, [0034], “Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”) having stored thereon instructions for performing a method (Gladilin, Fig. 4) comprising machine executable code which when executed by at least one machine (Aragaki, [0034], “As shown in FIG. 1, the image processing device 5 includes a normalizing unit 7, a reducing unit 9, a first smoothing processor (smoothing unit) 11, a seed creating unit (binarizing unit) 13, a grayscale correcting unit 15, a second smoothing processor (smoothing unit) 17, a schematic-shape-region creating unit 19, a region-expansion reference-image creating unit 21, a region expanding unit (region segmenting unit) 23, an enlarging unit 25, a filter (measuring unit, filtering unit) 27, and an output unit 29. These units are connected to and operationally controlled by a system controller (not shown). Moreover, these units may be constituted of, for example, a central processing unit (CPU) and a storage device that stores a computational program therein, such as a random access memory (RAM) and a read-only memory (ROM). In this case, the ROM may store a cell-cluster recognition program as the computational program.”). The proposed combination as well as the motivation for combining the Gladilin, Aragaki, Rad, and Datta references presented in the rejection of Claim 8, apply to Claim 24 and are incorporated herein by reference. Thus, the medium recited in Claim 24 is met by Gladilin, Aragaki, Rad, and Datta.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DENISE G ALFONSO whose telephone number is (571)272-1360. The examiner can normally be reached Monday - Friday 7:30 - 5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DENISE G ALFONSO/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662