DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a 2nd non-final rejection.
Priority
Receipt is acknowledged that application claims priority to foreign application with application number KR10-2022-0049149 dated 04/20/2022. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Response to Arguments
The claim objections and claim rejections under 35 U.S.C. 112(b) have been withdrawn in light of the amended claims.
The claim rejection under 35 U.S.C. 103 have been withdrawn in light of the Applicant’s arguments and a new rejection has been proposed.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 7, 12-14, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (US 20120134588 A1) in view of Laaksonen (US 20210065360 A1) and Cole (US 20190147642 A1).
Regarding claim 1, Zhang discloses a method, performed by an electronic device, of processing an image (Zhang, paragraph [0038], " The image or video 205 is either pre-recorded, or is recorded or captured using a conventional image or video capture device 210"), the method comprising:
obtaining a first image of a three-dimensional (3D) object comprising at least one surface by using a first camera, the at least one surface having a non-flat shape (Zhang, paragraph [0139], Fig. 7 below, "In general, as illustrated by FIG. 10, the Text Rectifier begins operation by receiving 1010 or selecting an image from some input source (e.g., a camera or database of images)").
PNG
media_image1.png
651
473
media_image1.png
Greyscale
While Zhang, teaches identifying a region of interest (ROI) within the first image of the 3D object, which corresponds to the at least one surface, as a region of interest (Zhang, paragraph [0139], Fig. 7 above black box, "Once an image has been received 1010 or selected, some region of that input image containing deformed or transformed text is then selected 1020 or otherwise designated by the user."), Zhang does not teach this “by applying the first image of the 3D object to a first artificial intelligence (AI) model”.
However, Laaksonen teaches identifying a region of interest (ROI) within the first image of the 3D object, which corresponds to the at least one surface by applying the first image of the 3D object to a first artificial intelligence (Al) model (Laaksonen, paragraph [0012], "Each first neural network model can be trained to approximate a region-of-interest (ROI) around an anatomical structure or a cut-off plane with respect to the anatomical structure").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to use Laaksonen’s AI model to identify a ROI from Zhang’s image.
The suggestion/motivation for doing so would have been to automate the process of identification and reduce human errors.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
While Zhang in view of Laaksonen teaches obtaining data about a 3D shape type of the 3d object (Zhang, paragraph [0088], "This leads to a very efficient and effective algorithm that can accurately recover the 3D shape and 2D texture of the surface as well as the camera pose (i.e., viewing direction, field of view, focal point, etc.) from a single image"), Zhang in view of Laaksonen does not teach this by applying the first image of the 3d object to a second Al model.
However, Cole teaches obtaining data about a 3D shape type of the 3d object by applying the first image of the 3d object to a second Al model (Cole, paragraph [0023], “FIG. 2A depicts an example system 200 for training a 3D estimator neural network 220 (such as the 3D estimator neural network 120 of FIG. 1) to estimate 3D shapes and textures of objects depicted in images with improved accuracy”).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to obtain data about the shape of Zhang’s (in view of Laaksonen) object by inputting Zhang’s images into an AI model, as taught by Cole.
The suggestion/motivation for doing so would have been to automate the process of data acquisition and reduce human errors.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Zhang in view of Laaksonen and Cole discloses obtaining a set of values of a 3D parameter related to at least one of the 3D object, the at least one surface, or the first camera, based on the region identified as the ROI and the data about the 3D shape type (Zhang, paragraph [0088], "This leads to a very efficient and effective algorithm that can accurately recover the 3D shape and 2D texture of the surface as well as the camera pose (i.e., viewing direction, field of view, focal point, etc.) from a single image."),
estimating the non-flat shape of the at least one surface, based on the set of values of the 3D parameter (Zhang, paragraph [0140], "Next, the Text Rectifier iteratively processes the selected region using an iterative convex optimization process 1030 that models the selected region as a low-rank matrix and repeatedly estimates and updates transforms that would cause the deformation of the text in the modeled region until convergence to a minimum rank of the matrix is achieved."),
and obtaining a flat surface image in which the non-flat shape of the at least one surface is flattened, by performing a perspective transformation on the at least one surface (Zhang, paragraph [0143], Fig. 7 below bottom image, "The Text Rectifier then outputs 1060 the low-rank texture version of the input image region for whatever further use or processing. For example, one such use is to optionally recognize 1070 text in the low-rank texture version of the input image region using an OCR engine.").
PNG
media_image1.png
651
473
media_image1.png
Greyscale
Therefore, it would have been obvious to combine Zhang in view of Laaksonen and Cole to obtain the invention as specified in claim 1.
Regarding claim 2, Zhang in view of Laaksonen and Cole discloses the method of claim 1, wherein the set of values of the 3D parameter comprises at least one of: a height value related to a 3D shape of the 3d object, a radius value related to the 3D shape of the 3d object an angle value of the ROI of the at least one surface of the 3d object, a translation value for 3D geometric transformation, a rotation value for the 3D geometric transformation, or a focal length value of the first camera (Zhang, paragraph [0088], "This leads to a very efficient and effective algorithm that can accurately recover the 3D shape and 2D texture of the surface as well as the camera pose (i.e., viewing direction, field of view, focal point, etc.) from a single image").
Regarding claim 3, Zhang in view of Laaksonen and Cole discloses the method of claim 1, wherein the first Al model is trained to infer a region corresponding to a surface in an image as an ROI (Laaksonen, paragraph [0010], "Each first neural network model can be trained to approximate the ROI or the cut-off plane"),
and wherein the second Al model trained to infer the 3D shape type of the 3d object in the image (Cole, paragraph [0023], “FIG. 2A depicts an example system 200 for training a 3D estimator neural network 220 (such as the 3D estimator neural network 120 of FIG. 1) to estimate 3D shapes and textures of objects depicted in images with improved accuracy”).
Regarding claim 7, Zhang in view of Laaksonen and Cole discloses the method of claim 1, further comprising obtaining information related to the 3D object from the flat surface image, wherein the obtaining of information related to the 3D object from the flat surface image comprises applying optical character recognition (OCR) to the flat surface image (Zhang, paragraph [0030], "Once distortions have been removed and the text or characters rectified, the resulting text is made available for a variety of uses or further processing such as optical character recognition (OCR).").
Claims 12-14 and 18 correspond to claims 1-3 and 7 respectively, additionally reciting a first camera (Zhang, paragraph [0058], “Consequently, given a curved surface, the modified TILT process will return the low-rank solution (i.e., I.sup.0, .tau., and E) for selected image regions, as well as the 3D curve geometry (C) and a corresponding rotation-translation pair (R,T) for the curved surface relative to the camera or image capture device used to capture the image”),
a memory storing one or more instructions (Zhang, paragraph [0150], “Computer storage media includes, but is not limited to, computer or machine readable media or storage devices such as DVD's, CD's, floppy disks, tape drives, hard drives, optical drives, solid state memory devices, RAM, ROM, EEPROM, flash memory or other memory technology, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other device which can be used to store the desired information and which can be accessed by one or more computing devices”),
and one or more processors configured to execute the one or more instructions stored in the memory, wherein the one or more processors is configured to execute the one or more instructions (Zhang, paragraph [0154], “Still further, the aforementioned instructions may be implemented, in part or in whole, as hardware logic circuits, which may or may not include a processor”). Thus, claims 12-14 and 18 are rejected for the same reasons of obviousness as claims 1-3 and 7.
Regarding claim 20, Zhang in view of Laaksonen and Cole discloses a non-transitory computer-readable recording medium having recorded thereon a computer program, which, when executed by a computer, performs the method of claim1 (Zhang, paragraph [0118], “Fortunately, since almost all digital cameras save JPEG files with EXIF (Exchangeable Image File) data, camera settings and scene information are recorded by the camera into the image file”).
Claim(s) 4 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (US 20120134588 A1) in view of Laaksonen (US 20210065360), Cole (US 20190147642 A1), and in further view of Kulkarni (US 20180232568 A1).
Regarding claim 4, Zhang in view of Laaksonen and Cole discloses the method of claim 1.
Zhang in view of Laaksonen and Cole does not teach “wherein the obtaining of the data about the 3D shape type of the 3d object comprises: receiving an user's input related to the 3D shape type of the 3d object from the user; and identifying the 3D shape type of the 3d object by applying a weight to a 3D shape type corresponding to the user's input among a plurality of 3D shape types”.
However, Kulkarni teaches wherein the obtaining of the data about the 3D shape type of the 3d object comprises: receiving an user's input related to the 3D shape type of the 3d object from the user; and identifying the 3D shape type of the 3d object by applying a weight to a 3D shape type corresponding to the user's input among a plurality of 3D shape types (Kulkarni, paragraph [0042], “The prediction module 213 receives the behavior of the user from the identification module 213 and the weights assigned to each input among the first set of inputs from the assigning module 211. Then, the prediction module 213 correlates inputs with higher weight and behavior of the user identified from the input. In one embodiment, the prediction module 213 may predict the interest of the user based on the behavior identified using the input assigned with higher value of weight”).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to receive user input for the shape of Zhang’s (in view of Laaksonen and Cole) object, and predict the shape using the user input, as taught by Kulkarni.
The suggestion/motivation for doing so would have been to manually identify shapes that may not be recognizable autonomously.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to Zhang in view of Laaksonen, Cole, and in further view of Kulkarni to obtain the invention as specified in claim 4.
Claim 15 corresponds to claim 4, additionally reciting one or more processors further configured to execute the one or more instructions (Zhang, paragraph [0154], “Still further, the aforementioned instructions may be implemented, in part or in whole, as hardware logic circuits, which may or may not include a processor”). Thus, claim 15 is rejected for the same reasons of obviousness as claim 4.
Claim(s) 8-9, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (US 20120134588 A1) in view of Laaksonen (US 20210065360), Cole (US 20190147642 A1), and in further view of Tal (US 20220116742 A1).
Regarding claim 8, Zhang in view of Laaksonen and Cole discloses the method of claim 1.
Zhang in view of Laaksonen and Cole does not teach “further comprising obtaining a second image of the 3d object by using a second camera having a wider angle of view than the first camera”.
However, Tal teaches further comprising obtaining a second image of the 3d object by using a second camera having a wider angle of view than the first camera (Tal, paragraph [0035], "Some camera(s) 112 can be geared towards obtaining a wider field of view 105 whereas others can obtain narrower field of view 105").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to implement another camera with a wider FOV into Zhang’s (in view of Laaksonen and Cole) scenario, as taught by Tal, and input the images into Cole’s model.
The suggestion/motivation for doing so would have been to because a wider FOV will allow for wider pictures that can recognize more of the labels on curved surfaces.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Zhang in view of Laaksonen, Cole, and in further view of Tal to obtain the invention as specified in claim 8.
Regarding claim 9, Zhang in view of Laaksonen, Cole, and Tal discloses the method of claim 8, wherein the obtaining of the data about the 3D shape type of the 3d object comprises obtaining information related to the 3D shape type of the 3d object by applying the second image to the second Al model (Cole, paragraph [0023], “FIG. 2A depicts an example system 200 for training a 3D estimator neural network 220 (such as the 3D estimator neural network 120 of FIG. 1) to estimate 3D shapes and textures of objects depicted in images with improved accuracy”, the wider images can be used as input for Cole’s AI model).
Regarding claim 19, Zhang in view of Laaksonen and Cole discloses the electronic device of claim 12.
Zhang in view of Laaksonen and Cole does not teach “wherein the electronic device further comprises a second camera having a wider angle of view than the first camera, and wherein the one or more processors are further configured to execute the one or more instructions to: obtain a second image of the 3d object by using the second camera”.
However, Tal teaches wherein the electronic device further comprises a second camera having a wider angle of view than the first camera, and wherein the one or more processors are further configured to execute the one or more instructions to: obtain a second image of the 3d object by using the second camera (Tal, paragraph [0035], "Some camera(s) 112 can be geared towards obtaining a wider field of view 105 whereas others can obtain narrower field of view 105").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to implement another camera with a wider FOV into Zhang’s (in view of Laaksonen and Cole) capturing to obtain wider images, as taught by Tal, and input the images into Cole’s model.
The suggestion/motivation for doing so would have been to because a wider FOV will allow for wider pictures that can recognize more of the labels on curved surfaces.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Zhang in view of Laaksonen, Cole, and Tal discloses obtain information related to the 3D shape type of the 3d object by applying the second image to the second Al model (Cole, paragraph [0023], “FIG. 2A depicts an example system 200 for training a 3D estimator neural network 220 (such as the 3D estimator neural network 120 of FIG. 1) to estimate 3D shapes and textures of objects depicted in images with improved accuracy”).
Therefore, it would have been obvious to combine Zhang in view of Laaksonen, Cole, and in further view of Tal to obtain the invention as specified in claim 19.
Claim(s) 10 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang (US 20120134588 A1) in view of Laaksonen (US 20210065360), Cole (US 20190147642 A1), Tal (US 20220116742 A1), and in further view of Baumgartner (US 20200027237 A1).
Regarding claim 10, Zhang in view of Laaksonen, Cole, and Tal discloses the method of claim 8.
Zhang in view of Laaksonen, Cole, and Tal does not teach “obtaining confidence of the ROI by applying the first image by using the first camera to the first Al model; obtaining confidence of the 3D shape type of the 3d object by applying a second image by using the second camera to the second Al model”.
However, Tal additionally teaches obtaining confidence of the ROI by applying the first image by using the first camera to the first Al model; obtaining confidence of the 3D shape type of the 3d object by applying the second image by using the second camera to the second Al model (Tal, paragraph [0107], "A detector 902 can also provide a score, typically known as confidence, which represents how sure the neural network 905 is in the object 12 detection").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to calculate a confidence for both AI models of Zhang (in view of Laaksonen, Cole, and Tal), as additionally taught by Tal.
The suggestion/motivation for doing so would have been to quantify how likely the AI models predicted their respective purposes, allowing for more fine tuning and understanding of the models at hand.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Zhang in view of Laaksonen, Cole, and Tal does not teach “capturing the first image and the second image, based on respective threshold values of the confidence of the 3D shape type of the 3d object and the confidence of the ROI, respectively”.
However, Baumgartner teaches capturing the first image and the second image, based on respective threshold values of the confidence of the 3D shape type of the 3d object and the confidence of the ROI, respectively (Baumgartner, paragraph [0012], "In one embodiment, the method comprises storing at least one image frame when the confidence level for that image frame exceeds the threshold amount and associating that image frame with its specified element. Accordingly, image frames whose confidence level exceeds the threshold amount may be stored and associated with the specified element which is imaged within those image frames for future reference and/or selection").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to capture and store images of Zhang (in view of Laaksonen, Cole, and Tal) if the image exceeds a confidence threshold, as taught by Baumgartner.
The suggestion/motivation for doing so would have been to acquire higher quality images that result in better training input.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Zhang in view of Laaksonen, Cole, and in further view of Tal and Baumgartner to obtain the invention as specified in claim 10.
Claim(s) 11 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang (US 20120134588 A1) in view of Laaksonen (US 20210065360), Cole (US 20190147642 A1), Tal (US 20220116742 A1), Baumgartner (US 20200027237 A1), and in further view of Tanaka (US 20110222743 A1).
Regarding claim 11, Zhang in view of Laaksonen, Cole, Tal and Baumgartner discloses the method of claim 10.
Zhang in view of Laaksonen, Cole, Tal and Baumgartner does not teach “further comprising: searching for matching data in a database, based on the flat surface image or information obtained from the flat surface image”.
However, Tanaka teaches further comprising: searching for matching data in a database, based on the flat surface image or information obtained from the flat surface image (Tanaka, paragraph [0005], “In the face matching device, the face image including the face of the person is previously registered in a database, and an input face image obtained by the photographing is matched with the database”).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to match Zhang’s (in view of Laaksonen, Cole, Tal, and Baumgarneter) flattened image with an image in a database, as taught by Tanaka.
The suggestion/motivation for doing so would have been to see how closely the predicted flattened image resembles the actual flattened image, resulting in better feedback and input.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Zhang in view of Laaksonen, Cole, Tal Baumgartner, and Tanaka does not teach “and displaying a result of the searching for matching data in the database”.
However, Cole additionally teaches displaying a result of the searching for matching data in the database (Cole, paragraph [0089], “To provide for interaction with a user, embodiments may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user may provide input to the computer”).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to use Cole’s display to display the evaluation values of Zhang (in view of Laaksonen, Cole, Tal, Baumgartner, and Tanaka).
The suggestion/motivation for doing so would have been to provide a visualization of the image.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Zhang in view of Laaksonen, Tal, Baumgartner, and in further view of Cole and Tanaka to obtain the invention as specified in claim 11.
Allowable Subject Matter
Claim 5-6, 16-17 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WAYNE ZHANG whose telephone number is (571) 272-0245. The examiner can normally be reached Monday-Friday 9:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ms. Sumati Lefkowitz can be reached on (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WAYNE ZHANG/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672