DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3 and 12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 3 and 12 recites the limitation " recognize a portion of the first input image with the texture intensity satisfying a second condition" in line 3. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 6, 8, 10-11, 15 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Narasimha (US 20170278231) in view of Lin (US 20200334854).
Regarding claim 1:
Narasimha teaches: a method for determining camera attitude (¶ [0001] “An embodiment of the invention relates generally to image processing, computer vision and robotics devices, and in particular, to a device for and a method of determining a pose of a camera.”); comprising:
in response to a first input image satisfying a first condition, adjusting a camera from a first attitude to a second attitude (¶ [0021] “The methods and devices set forth below provide camera pose failure detection and re-localization of a camera using a sensor fusion approach, and includes a user interface to enable re-localization. When performing frame-to-frame tracking in computer vision or SLAM systems, the estimated camera pose may not be reliable. For example, the number of features in a scene may be too low to obtain a good camera pose estimate [first condition], such as due to flat regions (e.g. a wall), poor lighting, motion blur/focus issues, or fast camera motion or motion caused by handshake.”; ¶ [0033] “…The user interface 325 provides various user guidance, such as prompts to a user of the camera to move the camera in a certain direction to effectively continue the 3D scanning operation. For example, the user interface may prompt a user of the camera to move the camera to the left” Note that the instant application in ¶ [0068] discloses “ the camera attitude may be adjusted through user's neck rotation”; therefore, providing prompts to a user of the camera to move the camera in a certain direction satisfies the limitation under BRI);
the first input image being an image obtained by the camera in the first attitude (¶ [0027] “Turning now to FIG. 3, a diagram shows blocks for detecting a camera pose failure. A first frame 302, designated as Frame n, and a second frame 304, designated as Frame n+1, comprise different components including RGB data, IMU data, depth data, or other sensor data that are coupled to a plurality of processing blocks”);
obtaining a second input image, the second input image being an image obtained by the camera in the second attitude (¶ [0026] “Two aspects of the circuits and methods include the detection of camera pose failure and the re-localization of the camera which is tracking the subsequent frames to stored key frames until reliable features are obtained (and a key frame to which the incoming frame is localized is determined”; ¶ [0031] “…If the number of matches is below the threshold value, a camera pose failure flag is set and the algorithm stops tracking to the previous frame. The camera then switches to matching the incoming frames to all the key frames so that a match can be found to resume the tracking/matching process”);
determining second attitude data corresponding to the second attitude based on the second input image (¶ [0026] “Two aspects of the circuits and methods include the detection of camera pose failure and the re-localization of the camera which is tracking the subsequent frames to stored key frames until reliable features are obtained (and a key frame to which the incoming frame is localized is determined”; ¶ [0030] “…”A two-dimensional (2D) pose obtained using the RGB frames localizes the camera in the world co-ordinates, which is also known as visual odometry (VO).”; ¶ [0031] “…If the number of matches is below the threshold value, a camera pose failure flag is set and the algorithm stops tracking to the previous frame. The camera then switches to matching the incoming frames to all the key frames so that a match can be found to resume the tracking/matching process”; ¶ [0032] “…The camera pose failure detection block 314 also controls a camera re-localization block 322, as shown and described in more detail in reference to FIG. 4”);
While Narasimha discloses in ¶ [0037] “When performing feature tracking or matching, a feature detection algorithm may fail due to number of reasons such as motion blur/hand shake, limited objects or features to compute reliable camera pose, camera obstruction, or the dropping of a camera, for example. In these scenarios, the camera has to be re-localized to the previously seen landmarks or a 3D map that is stored in order to continue from where it was lost.”
Narasimha does not specifically teach: determining first attitude data corresponding to the first attitude according to a transformation relationship between the first attitude and the second attitude, and the second attitude data.
PNG
media_image1.png
138
500
media_image1.png
Greyscale
However, in a related field, Lin teaches: determining first attitude data corresponding to the first attitude according to a transformation relationship between the first attitude and the second attitude, and the second attitude data (¶ [0058] “305. Acquire position and attitude parameters of the image according to the position and attitude parameters of the image relative to the second marked image and position and attitude parameters of each marked image relative to a previous marked image, and determine a position and an attitude of the camera according to the position and attitude parameters”; ¶ [0062] “…the position and attitude parameters of the first image are acquired according to the position and attitude parameters of the first image relative to the second marked image and the position and attitude parameters of the second marked image relative to the initial marked image by using the following formula: .
Note that in Lin ¶[0062]: first image parameters are acquired according to (i) first image relative second marked image and (ii) second marked image relative initial marked image (that is two pose relationships chained). Therefore, “second attitude data” (pose of second marked image relative initial), and “transformation relationship” (relative transform between first image and second marked image), and “first attitude data” (pose of first image relative initial).”).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Narasimha to incorporate the teachings of Lin by including: determining first attitude data corresponding to the first attitude according to a transformation relationship between the first attitude and the second attitude, and the second attitude data in order to improve pose continuity and pose bookkeeping during recovery. a person of ordinary skill in the art prior would compute the first pose: e.g., continuity of pose chain/filling the gap between the last reliable pose and the re-localized pose.
Regarding claim 2:
Narasimha in view of Lin teaches the limitations of claim 1 as applied above.
Narasimha further teaches: wherein the first input image satisfying the first condition includes at least one of: a texture intensity of the first input image not reaching a preset value; failing to calculate the first attitude data through the first input image; or a pedestrian area interference value appearing in the first input image (¶ [0021] “The methods and devices set forth below provide camera pose failure detection and re-localization of a camera using a sensor fusion approach, and includes a user interface to enable re-localization. When performing frame-to-frame tracking in computer vision or SLAM systems, the estimated camera pose may not be reliable. For example, the number of features in a scene may be too low to obtain a good camera pose estimate [first condition], such as due to flat regions (e.g. a wall) [texture], poor lighting, motion blur/focus issues, or fast camera motion or motion caused by handshake.”).
Regarding claim 6:
Narasimha in view of Lin teaches the limitations of claim 1 as applied above.
Narasimha further teaches: third attitude data being attitude data determined according to a third input image obtained by the camera in a third attitude (¶ [0036] “…the last tracked frame is stored as a key frame along with the edge map, the depth frame, feature descriptors, the pose computed after fusion from RGB tracking/matching, depth alignment, and the prediction model output.” Therefore, the tracked frame is an image captured the camera at some camera attitude (third attitude), and Narasimha stores pose with that image, which corresponds to determining third attitude data according to a third input image);
Lin further teaches: wherein determining the first attitude data corresponding to the first attitude according to the transformation relationship between the first attitude and the second attitude, and the second attitude data comprises:
determining the first attitude data corresponding to the first attitude according to the transformation relationship between the first attitude and the second attitude, the second attitude data, and third attitude data (¶ [0058] “305. Acquire position and attitude parameters of the image according to the position and attitude parameters of the image relative to the second marked image and position and attitude parameters of each marked image relative to a previous marked image”; ¶ [0062] “…the position and attitude parameters of the first image are acquired according to the position and attitude parameters of the first image relative to the second marked image and the position and attitude parameters of the second marked image relative to the initial marked image by using the following formula: .
PNG
media_image1.png
138
500
media_image1.png
Greyscale
R_old, T_old represent the third attitude data.
).
Regarding claim 8:
Narasimha in view of Lin teaches the limitations of claim 7 as applied above.
Narasimha further teaches: wherein determining the first attitude data corresponding to the first attitude according to the transformation relationship between the first attitude and the second attitude, the second attitude data, and the third attitude data comprises:
determining the first attitude data corresponding to the first attitude according to the transformation relationship between the first attitude and the second attitude, the second attitude data, the transformation relationship between the first attitude and the third attitude, and the third attitude data (¶ [0058] “305. Acquire position and attitude parameters of the image according to the position and attitude parameters of the image relative to the second marked image and position and attitude parameters of each marked image relative to a previous marked image”; ¶ [0062] “…the position and attitude parameters of the first image are acquired according to the position and attitude parameters of the first image relative to the second marked image and the position and attitude parameters of the second marked image relative to the initial marked image by using the following formula: .
PNG
media_image1.png
138
500
media_image1.png
Greyscale
R_old, T_old represent the third attitude data. ¶ [0063] “R_final representing a rotation parameter of the first image, and T_final representing a translation parameter of the first image; Rcl representing a rotation parameter of the first image relative to the second marked image, and Tcl representing a translation parameter of the first image relative to the second marked image; and R_old representing a rotation parameter of the second marked image relative to the initial marked image, and T_old representing a translation parameter of the second marked image relative to the initial marked image.”).
Regarding claims 10-11, 15 and 17: the claims limitations are similar to those of claims 1-2, 6, and 8; therefore, rejected in the same manner as applied above. Narasimha discloses an electronic device in FIG. 2
Claim(s) 3, 9, 12, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Narasimha (US 20170278231) in view of Lin (US 20200334854) and Masad (US 20170208243).
Regarding claim 3:
Narasimha in view of Lin teaches the limitations of claim 1 as applied above.
Narasimha further teaches: wherein adjusting the camera from the first attitude to the second attitude comprises: recognizing a portion of the first input image with the texture intensity satisfying a second condition (¶ [0022] “…An edge map of the foreground silhouette from the last key-frame can also be used to align with an object for fast snapping to provide the user with an intuitive experience to re-localize the camera in a power efficient manner”; ¶ [0048] “…It is then determined whether a fast motion scenario is detected at a block 1004. If so, an edge map of a foreground silhouette is used for the last key frame to align with an object at a block 1006. Information is provided on a user interface of the camera to aid in the alignment at a block 1008.”);
adjusting the camera from the first attitude to the second attitude (¶ [0033] “…the user interface may prompt a user of the camera to move the camera to the left”);
Narasimha does not specifically teach: determining a first adjustment parameter according to a position of the portion in the first input image.
However, in a related field, Masad teaches: determining a first adjustment parameter according to a position of the portion in the first input image (¶ [0089] “…the instruction to the user may be determined by defining a vector of a desired movement by the difference between the maximum point of the sum-function and the center of the image”)and
adjusting the camera from the first attitude to the second attitude according to the first adjustment parameter (¶ [0090] “…the instructions may be displayed to the user as visual graphic marks overlaid on the image, such as an arrow pointing towards a desired direction of camera movement or arrows pointing in or out to indicate zoom in or zoom out actions and/or as text (i.e., written instructions). Alternatively or additionally, the instructions may be verbal, audibly provided by a speaker”).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Narasimha in view of Lin to incorporate the teachings of Masad by including: determining a first adjustment parameter according to a position of the portion in the first input image in order to provide a predictable results of a more clearly defined reproducible movement prompt.
Regarding claim 9:
Narasimha in view of Lin teaches the limitations of claim 1 as applied above.
Narasimha further teaches: sending a display instruction to a control terminal (¶ [0024] “…The processor circuit could be coupled to a display 206 for displaying a captured image, and more particularly, displaying frames of a 3D tracking application.”; ¶ [0033] “A user interface control block 324 also receives control signals from the camera pose failure detection block 314 and controls a user interface 325.”);
While Narasimha teaches in ¶ [0033] “…The user interface 325 provides various user guidance, such as prompts to a user of the camera to move the camera in a certain direction to effectively continue the 3D scanning operation. For example, the user interface may prompt a user of the camera to move the camera to the left”.
Narasimha does not specifically teach: the display instruction being used to instruct the control terminal to display a direction prompt icon, and a direction indicated by the direction prompt icon being consistent with a direction in which the camera is adjusted from the first attitude to the second attitude.
However, Masad further teaches: the display instruction being used to instruct the control terminal to display a direction prompt icon and a direction indicated by the direction prompt icon being consistent with a direction in which the camera is adjusted from the first attitude to the second attitude (¶ [0090] “…the instructions may be displayed to the user as visual graphic marks overlaid on the image, such as an arrow pointing towards a desired direction of camera movement or arrows pointing in or out to indicate zoom in or zoom out actions and/or as text (i.e., written instructions)”).
Regarding claims 12 and 18 : the claims limitations are similar to those of claims 3 and 9; therefore, rejected in the same manner as applied above.
Claim(s) 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Narasimha (US 20170278231) in view of Lin (US 20200334854) and Shin (US 20160010982).
Regarding claim 4:
Narasimha in view of Lin teaches the limitations of claim 1 as applied above.
Lin further teaches: the transformation relationship being a preset transformation relationship corresponding to the preset angle (¶ [0102] “3. Acquire, through a module 608 in combination with data in the IMU queue and data obtained by tracking a feature point, a translation parameter and a rotation parameter, and calculate a rotation and translation matrix from a current image relative to a currently marked image.”
Narsimha in view of Lin does not specifically teach: wherein adjusting the camera from the first attitude to the second attitude comprises: rotating the camera to a preset angle.
However, in a related field, Shin teaches: wherein adjusting the camera from the first attitude to the second attitude comprises:
rotating the camera to a preset angle (FIG. 6B, ¶ [0062] “…The camera 202 and laser projector 602 are fixed on a motorized rotation stage 654 configured to rotate the camera 202 and laser projector 602 about a common vertical axis.”; ¶ [0063] “…the motorized rotation stage 654 has a measurement precision of 1 degree and a higher precision for actuation”)).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Narasimha in view of Lin to incorporate the teachings of Shin by including: rotating the camera to a preset angle in order to support the camera pose recovery.
Regarding claim 13 : the claims limitations are similar to those of claim 4; therefore, rejected in the same manner as applied above.
Claim(s) 5 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Narasimha (US 20170278231) in view of Lin (US 20200334854) and Liang (US 20180324345).
Regarding claim 5:
Narasimha in view of Lin teaches the limitations of claim 1 as applied above.
Narsimha in view of Lin does not specifically teach: wherein adjusting the camera from the first attitude to the second attitude comprises: receiving a control instruction sent from a control terminal, and rotating the camera according to the control instruction until receiving a confirmation instruction sent from the control terminal.
However, in a related field, Liang teaches: wherein adjusting the camera from the first attitude to the second attitude comprises: receiving a control instruction sent from a control terminal (¶ [0015] “…transmit one or more search instructions for controlling the camera to perform a first rotation to the camera, until the target object is detected in the image transmitted by the camera in the first rotation”), and
rotating the camera according to the control instruction until receiving a confirmation instruction sent from the control terminal (¶ [0015] “…transmit one or more search instructions for controlling the camera to perform a first rotation to the camera, until the target object is detected in the image transmitted by the camera in the first rotation”; ¶ [0042] “…In a case that the camera control device detects the target object in a certain image, the camera control device may transmit an instruction to stop the rotation to the camera”).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Narasimha in view of Lin to incorporate the teachings of Liang by including: receiving a control instruction sent from a control terminal, and rotating the camera according to the control instruction until receiving a confirmation instruction sent from the control terminal in order to improve reliability and controllability of the re-localization maneuver.
Regarding claim 14 : the claims limitations are similar to those of claim 5; therefore, rejected in the same manner as applied above.
Allowable Subject Matter
Claims 7 and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Relevant art not relied upon
Dai (US 20190259176) teaches: a method and an apparatus for determining an attitude angle of a camera, capable of improving the accuracy of the attitude angle of the camera, such that a pose of the camera may be obtained more accurately based on the attitude angle of the camera. The present disclosure further provides a method and an apparatus for measuring a distance to an object, capable of measuring the distance to the object based on an attitude angle of a camera as determined in accordance with the above method for determining the attitude angle of the camera, such that the accuracy of the distance measurement may be improved. Finally, the present disclosure further provides a method and an apparatus for vehicle positioning, capable of determining a position of a vehicle based on an attitude angle of a camera as determined in accordance with the above method for determining the attitude angle of the camera, such that the accuracy of the vehicle positioning may be improved.
Zheng (US 20220398767) teaches: A pose is used for describing a position and an attitude of a camera. The position may be represented by coordinates of the camera in the world coordinate system. The attitude refers to an orientation of the camera in the world coordinate system, which may be represented by a rotation matrix.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WASSIM MAHROUKA whose telephone number is (571)272-2945. The examiner can normally be reached Monday-Thursday 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WASSIM MAHROUKA/Primary Examiner, Art Unit 2665