DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without “significantly more”. Claim(s) 1-20 is/are directed to Abstract Idea such as mathematical concept (i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations) for example organizing information and manipulating information through mathematical correlations.
The apparatus and the method claim 1, 8 and 14 recites limitation, “generating a compressed log depth map from a digital image utilizing a depth prediction machine learning model; converting the compressed log depth map to a depth map utilizing an exponential function; and generating a modified digital image from the digital image utilizing the depth map”. Since the claim is directed to a process and a machine, which is one of the statutory categories of the invention (Step 1: YES).
The claim is then analyzed to determine whether it is directed to any judicial exception. The claim recites generating step; converting step; and generating a modified digital image step which can be related to organizing information and manipulating information through mathematical correlations. Digitech Image Techs., LLC v. Electronics for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014). The patentee in Digitech claimed methods of generating first and second data by taking existing information i.e., generating a compressed log depth map from a digital image utilizing a depth prediction machine learning model, manipulating the data using mathematical functions i.e., converting the compressed log depth map to a depth map utilizing an exponential function, and organizing this information into a new form i.e., generating a modified digital image from the digital image utilizing the depth map. The court explained that such claims were directed to an abstract idea because they described a process of organizing information through mathematical correlations, like Flook's method of calculating using a mathematical formula. 758 F.3d at 1350, 111 USPQ2d at 1721. (Step 2A: Prong One Abstract Idea=Yes).
The claim is then analyzed if it requires an additional elements or a combination of additional elements in the claim to apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the exception – i.e., limitation that are indicative of integration into a practical application: improving to the functioning of a computer or to any other technology or technical field. In the current claims, there is no additional elements that would integrate the abstract idea into a practical application (Step 2A: Prong Two Abstract Idea=Yes).
Next the claim is analyzed to determine if there are additional limitation recited in the claim such that the claim amount to significantly more than an abstract idea. The claim requires the additional limitation of a computer with the central processing unit, memory, a printer, an input and output terminal and a program. These generic computer components are claimed to perform the basic functions of storing, retrieving and processing data through the program that enables. In the current scenario, there are no additional elements that would amount to significantly more than the abstract idea. Therefore, the claim does not amount to significantly more than the abstract idea itself (Step 2B: No). Accordingly, the claim is not patent eligible.
Further, dependent claims do not add any positive limitation or step that recite within the scope of the claim and do not carry patentable weight they are also rejected for the same reasons as independent claims.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claim 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1-20 of Patent No. US 12125227 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because all the claimed limitations recited in pending application are transparently found in Patent No. US 12125227 B2 with obvious wording variation. For example, compare Claim 1 of pending application with claim 18 and 19 of Patent No. US 12125227 B2, they both recite
A computer-implemented method comprising (A computer-implemented method comprising):
generating a compressed log depth map from a digital image utilizing a depth prediction machine learning model (generating a predicted depth map for a digital image utilizing a depth prediction machine learning model);
converting the compressed log depth map to a depth map utilizing an exponential function (claim 19, converting the compressed log depth map to a depth map utilizing an exponential function); and
generating a modified digital image from the digital image utilizing the depth map (claim 19, generating a modified digital image from the additional digital image utilizing the depth map),
Further, analyzing and comparing dependent claims 2-7 of the pending application with claims 19-20 of Patent No. US 12125227 B2 it was found that they recite the same limitation with wording changes.
Similarly, analyzing and comparing independent claims 8 and 14 of the pending application including its dependent claims with claims 1 and 9 including its dependent claims of Patent No. US 12125227 B2 it was found that they recite the same limitation with wording changes.
Note the claims issued of Patent No. US 12125227 B2 are narrower in scope such that the claimed limitations as recited in pending application are encompassed by Patent No. US 12125227 B2.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 5-7, 14, 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hyungjoo Jung title DEPTH PREDICTION FROM A SINGLE IMAGE WITH CONDITIONAL ADVERSARIAL NETWORKS 2017.
Regarding Claim 1. Jung teaches a computer-implemented method (Fig. 2 and Page 1718 Sec 2.2 network architecture) comprising:
generating a compressed log depth map (Page 1718 Sec 2.2.1, Generator network, The generator G implicitly defines a probability distribution pG as the distribution of the samples G(I). If pG and pgt are concentrated on very different manifolds, the discrimination problem between them becomes trivial. In this case, log(1 − D((I,G(I)))) is saturated regardless of the result of G (the left part of Fig. 1) i.e., generating a compressed log depth map) from a digital image (Page 1718 Sec 2.2.1, Generator network, It takes the single image I as an input and generates the initial depth image i.e. from a digital image) utilizing a depth prediction machine learning model (we gradually reduce the spatial resolution with 2 × 2 maxpooling (stride 2) while doubling the number of channels. The decoder part predicts the initial depth estimate through a sequence of deconvolutional (a factor of 2) and convolutional layers i.e., utilizing a depth prediction machine learning model) ;
converting the compressed log depth map to a depth map utilizing an exponential function (Page 1719 Sec 2.3, Learning Adversarial Network, We train our model in two phases using stochastic gradient method (SGM): The global net is first pre-trained, and these parameters are fixed during learning the refinement net and discriminator. The convolutional layers in the global net are initialized using the Oxford VGG-net [22]. GivenM training RGB-D pairs {I(p), d(p)}Mp=1, we apply the L1 loss directly for the global net:
Lg =1/M Ep_u(p)g− d(p)_1 , (2)
where ug denotes the output of global net. Though L1 (or L2) loss tends to produce blurry estimates on generation problems, this content loss captures the low frequency information reliably in many tasks. After pre-training the global net, we jointly train the refinement net and the discriminator to capture high-frequency details by solving. The min-max problem is solved by alternatively applying gradient descent and gradient accent once. The discriminator D is trained by maximizing (1) with fixed G. The minimization problem in only updates the parameters of refinement net by minimizing the following loss:
Lr =1/M E_p__u(p)r− d(p)_1 + μ log(1 − D((I(p), u(p) r ))) (3)
where ur is the output of refinement net (μ = 1/λ). The adversarial loss, log(1 − D((I, ur))) i..e, converting the compressed log depth map to a depth map utilizing a function;
and
generating a modified digital image from the digital image utilizing the depth map ((Page 1719 Sec 2.3, Learning Adversarial Network, encourages our model to generate more natural and structure preserving depth predictions i.e., generating a modified digital image from the digital image utilizing the depth map).
Jung teaches utlizing loss function instead of exponential function.
However, it is well known in the art that loss function is a mathematical function that quantifies the difference between a model’s predicted output and the true target value. It is not inherently an exponential function, but some loss functions are defined using exponential function and therefore, loss function can be treated as exponential function (See definition for loss function and exponential function).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Jung so as to enable more realistic and structure-preserving depth prediction from single image thus having clear boundary localization and spurious regions thus resulting depth images that are visually satisfied (See Jung Abstract).
Regarding Claim 5, Jung teaches wherein generating the modified digital image from the digital image utilizing the depth map comprises utilizing the depth map with a filter model to generate the modified digital image (Page 1717 Col 2 Para 2 L 18-24).
Regarding Claim 6, Jung teaches wherein generating the modified digital image from the digital image utilizing the depth map comprises utilizing the depth map to generate a blurred background of the modified digital image from the digital image (Page 1719 Sec 2.3 Col 1 Para 2).
Regarding Claim 7, Jung teaches wherein the depth prediction machine learning model is trained to generate compressed log depth maps from training digital images and compressed log ground truth depth maps (Fig. 3 and also see Sec 2.3 Learning Adversaries network Col 1 Para 2).
Regarding Claim 14, it has been rejected for the same reasons as claim 1.
Regarding Claim 18, it has been rejected for the same reasons as claim 5.
Regarding Claim 19, it has been rejected for the same reasons as claim 6.
Regarding Claim 20, it has been rejected for the same reasons as claim 7.
Claim(s) 2-4, 8-11, 13, 15-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hyungjoo Jung title DEPTH PREDICTION FROM A SINGLE IMAGE WITH CONDITIONAL ADVERSARIAL NETWORKS 2017 and further in view of Anisimovskiy et al. Pub. No. US 20200074661 A1
Regarding Claim 2, Jung does not specifically teach wherein converting the compressed log depth map to the depth map utilizing the exponential function further comprises converting the compressed log depth map to a disparity map utilizing the exponential function.
However, in the same field of endeavor, Anisimovskiy teaches that The original or corrected left and right images I.sub.L and I.sub.R, reconstructed left and right images I′.sub.L and I′.sub.R, disparity maps d.sub.L and d.sub.R, reconstructed disparity maps d′.sub.L, and d′.sub.R for left and right images and the auxiliary images I″.sub.L and I″.sub.R are further used to form a loss function, which is used as a training signal for the entire convolutional neural network to reconstruct the analyzed image (i.e., the formed loss function is used to train the entire network (selection of network weights)) in accordance with the back propagation method. In particular, to form a loss function, the reconstructed images I′.sub.L and I′.sub.R and the auxiliary images I″.sub.L and I″.sub.R are compared with the original or corrected images I.sub.L and I.sub.R, and the reconstructed disparity maps d′.sub.L and d′.sub.R are compared with the originally produced disparity maps d.sub.L and d.sub.R. As a result, the trained siamese subnetwork is capable of depth map estimation from a single image. It is preferred to use the corrected images with the help of the generated left and right image correction maps to produce the reconstructed images I′.sub.L and I′.sub.R and auxiliary images I″.sub.L and I″.sub.R i.e., converting the compressed log depth map to the depth map utilizing the exponential function further comprises converting the compressed log depth map to a disparity map utilizing the exponential function (Para 39).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Jung with the method of Anisimovskiy so as to improve precision when estimating a depth map using a trained Siamese subnetwork (See Anisimovskiy Para 39).
Regarding Claim 3, Jung does not specifically teach wherein converting the compressed log depth map to the depth map utilizing the exponential function further comprises converting the disparity map to the depth map utilizing an inverse function.
However, in the same field of endeavor, Anisimovskiy teaches original input image was an arbitrary image of a street, on which a lot of cars are parked and cyclists pass, and there are buildings nearby. Above the original input image is an inverse depth map, produced with respect to the original input image in accordance with the described method. The following image, located above the inverse depth map, is the corresponding disparity map. Both the inverse depth map and the disparity map depict obviously the main objects captured in the original input image i.e., converting the compressed log depth map to the depth map utilizing the exponential function further comprises converting the disparity map to the depth map utilizing an inverse function (Para 41).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Jung with the method of Anisimovskiy so that the errors are smoothed and the precision of the depth map estimation is significantly improved.
Regarding Claim 4, Jung does not specifically teach further comprising determining an exponent for the exponential function based on one or more distance distribution metrics of one or more digital images.
However, in the same field of endeavor, Anisimovskiy teaches taking stereo pair images including a left and right images (I.sub.L, I.sub.R), inputting each of the left and right images (I.sub.L, I.sub.R) to a corresponding siamese convolutional neural network for depth map estimation, processing the input left and right images (I.sub.L, I.sub.R) to produce high-level feature maps and inverse depth maps for the left and right images, respectively, inputting the produced high-level feature maps to the convolutional neural network for camera parameters estimation, processing the produced high-level feature maps to produce parameters of a camera which shot the left and right images (I.sub.L, I.sub.R), applying an affine transform to the inverse depth maps for the left and right images (I.sub.L, I.sub.R), respectively, taking into account the produced camera parameters to produce disparity maps (d.sub.L, d.sub.R) for the left and right images (I.sub.L, I.sub.R), respectively, performing bilinear-interpolation sampling for the left image I.sub.L taking into account the produced disparity map d.sub.R for the right image I.sub.R to produce a reconstructed right image I′.sub.R, performing bilinear-interpolation sampling for the right image I.sub.R taking into account the produced disparity map d.sub.L for the left image I.sub.L to produce a reconstructed left image I′.sub.L, performing bilinear-interpolation sampling for the disparity map d.sub.L for the left image I.sub.L taking into account the produced disparity map d.sub.R for the right image I.sub.R to produce a reconstructed disparity map for the right image I.sub.R, performing bilinear-interpolation sampling for the disparity map d.sub.R for the right image I.sub.R taking into account the produced disparity map d.sub.L for the left image I.sub.L to produce a reconstructed disparity map for the left image I.sub.L, performing bilinear-interpolation sampling for the left image I.sub.L taking into account the reconstructed disparity map d′.sub.R for the right image I.sub.R to get an auxiliary right image I″.sub.R, performing bilinear-interpolation sampling for the right image I.sub.R taking into account the reconstructed disparity map d′.sub.L for the left image I.sub.L to get an auxiliary left image I″.sub.L, forming a common loss function basing on the left and right images (I.sub.L, I.sub.R), reconstructed left and right images (I′.sub.L, I′.sub.R), disparity maps (d.sub.L, d.sub.R), reconstructed disparity maps (d′.sub.L, d′.sub.R) for the left and right images (I.sub.L, I.sub.R) and the auxiliary images (I″.sub.L, I″.sub.R), and training the neural network based on the formed loss function where generating disparity maps, an additional loss function is calculated by taking into account the maximum distance heuristic (MDH) as in Equation (Para 8 and 69).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Jung with the method of Anisimovskiy so that the errors are smoothed and the precision of the depth map estimation is significantly improved.
Regarding Claim 8, it has been rejected for the same reasons as claim 1 and 3 and further teaches a system (fig. 2 Page 1717 Sec 1 Introduction Para 1 human visual system) comprising: one or more memory devices; and one or more processors coupled to the one or more memory devices (fig. 2 Page 1717 Sec 1 Introduction Para 1 human visual system inherently has a memory and process to cause to system to perform).
Regarding Claim 9, it has been rejected for the same reasons as claim 1.
Regarding Claim 10, it has been rejected for the same reasons as claim 5 or 6.
Regarding Claim 11, it has been rejected for the same reasons as claim 4.
Regarding Claim 13, it has been rejected for the same reasons as claim 7.
Regarding Claim 15, it has been rejected for the same reasons as claim 2.
Regarding Claim 16, it has been rejected for the same reasons as claim 3.
Regarding Claim 17, it has been rejected for the same reasons as claim 4.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hyungjoo Jung title DEPTH PREDICTION FROM A SINGLE IMAGE WITH CONDITIONAL ADVERSARIAL NETWORKS 2017 and further in view of Anisimovskiy et al. Pub. No. US 20200074661 A1 and further in view of Niewczar et al. Pub. No. US 20200272865 A1
Regarding Claim 12, Jung and Anisimovskiy does not specifically teach wherein the operations further comprise determining the one or more distance distribution metrics by determining a distance mean and standard deviation of the one or more digital images.
However, in the same field of endeavor, Niewczar teaches from Fig. 5 that the quality of a classification or cluster of features is further characterized by determining a distance metric for each feature in the classification, where the distance metric indicates the deviation of the feature from mean cluster image (Para 29).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to combine the method of Jung with the method of Anisimovskiy and further with the method of Niewczar so as to correct different errors and improve future classification (See Niewczar Para 29).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zhou et al. Pub. No. US 20230236219 A1 - VISUAL INERTIAL ODOMETRY WITH MACHINE LEARNING DEPTH
Gao et al. Pub. No. US 20230072702 A1 - ACCELERATING SPECKLE IMAGE BLOCK MATCHING USING CONVOLUTION TECHNIQUES
Baig et al. Patent No. US 11238604 B1 - Densifying sparse depth maps
Ye et al. Pub. No. US 20210390339 A1 - DEPTH ESTIMATION AND COLOR CORRECTION METHOD FOR MONOCULAR UNDERWATER IMAGES BASED ON DEEP NEURAL NETWORK
Wang et al. Pub. No. US 20210174524 A1 - SYSTEMS AND METHODS FOR DEPTH ESTIMATION VIA AFFINITY LEARNED WITH CONVOLUTIONAL SPATIAL PROPAGATION NETWORKS
Dudzik et al. Pub. No. US 20210150278 A1 - DEPTH DATA MODEL TRAINING
Guizilini et al. Pub. No. US 20210004646 A1 - SYSTEMS AND METHODS FOR WEAKLY SUPERVISED TRAINING OF A MODEL FOR MONOCULAR DEPTH ESTIMATION
Kweon et al. Pub. No. US 20190385325 A1 - APPARATUS AND METHOD FOR DEPTH ESTIMATION BASED ON THERMAL IMAGE, AND NEURAL NETWORK LEARNING METHOD THEREFOF
CN 107403415 A - compression depth based on picture quality enhancement method and device of full convolutional neural network
CN 107358576 A - Depth image super-resolution reconstruction method based on convolutional neural network
David Elgen - Depth Map Prediction from a Single Image using a Multi-Scale Deep Network - 2014
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIZAR N SIVJI whose telephone number is (571)270-7462. The examiner can normally be reached Monday-Friday 7-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alison Slater can be reached at (571) 270-0375. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
NIZAR N. SIVJI
Primary Examiner
Art Unit 2647
/NIZAR N SIVJI/ Primary Examiner, Art Unit 2647
Prosecution Insights