Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 2-21 are rejected under 35 U.S.C. 103 as being unpatentable over PIRCHHEIM et al (Handling Pure Camera Rotation in Keyframe-Based SLAM) in view of PAPARI et al (Edge and line oriented contour detection: State of the art), RUZON et al (Edge, Junction, and Corner Detection Using Color Distributions), and KIM et al (US 20079914482)
As per claim 2, Pirchheim teaches the claimed “system” comprising: “one or more sensors for generating image data associated with a physical environment; a display; one or more processors; and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors” (Pirchheim, 6 RESULTS, The evaluation was performed on a laptop PC equipped with a quad-core 2.5GHz CPU and 8GB of RAM. We used a PointGrey Firefly MV handheld camera and recorded the images with a resolution of 640x480 pixels; 6.1 Mobile phone application - Our hybrid SLAM system also runs in real-time on modern mobile phones. We used a Samsung Galaxy S2 equipped with a dual-core 1.2GHz ARM CPU and 2GB RAM running Android OS 4 for our tests. As expected, we found the overall robustness and performance restricted in comparison to the PC version), cause the one or more processors to perform operations comprising: “generating, based at least in part on the image data, a point of interest map associated with the physical environment” (Pirchheim, 3.1 Incremental pose tracking - We select a feature set for matching from all map features by filtering features for (1) visibility from the predicted camera pose, (2) only infinite features of the currently enabled panorama map, (3) overlapping feature re-projections where we prefer finite over infinite features. At any point in time, either none or exactly one panorama map is enabled for tracking) (Noted: Pirchheim’s map features comprises point features as distinct visual or geometric landmarks that algorithms track across sequential frames). It is noted that Pirchheim does not teach, but Ruzon teaches “identifying, based at least in part on a contrast associated with a portion of the image data, at least a first line segment and a second line segment associated with the physical environment” (Ruzon, 4 EDGE DETECTION - Here, we present a method of estimating the strength and orientation of an edge hypothesized to split a window. To summarize, we divide the window in half with a line segment, compute a color signature for each half, and find the EMD between them. Repeating this process using line segments with different orientations reveals the true strength and orientation of the edge; 5 JUNCTION DETECTION - Fig. 6 shows different line segments representing edges); “forming a line associated with the physical environment based at least in part on the first line segment and the second line segment” (Ruzon, 5 JUNCTION DETECTION and 6 CORNER DETECTION – the detected junction or corners which are used to combine different detected edge segments; see also Kim, Figures 3E-3F - FIG. 3E shows a linked edge line formed by connecting edge point A and B according to an embodiment of the invention. The linked edge line has a slope S and the edge points A and B have directions DA and DB respectively. The linked edge line is formed due to DA and DB compatibility. However, if angle difference of slope S to DA or DB is significant, inconsistent data is generated in the missing block); and “projecting the line into a virtual scene representative of the physical environment” (Ruzon, 7.1 Edges and Junctions - Fig. 11 contains four regions, two light and two dark. The edge separating the two dark regions has relatively low contrast; see also Papari, 4.2. Grouping pixels into contours according to Gestalt principles, Figure 17 - Edge pixels can be grouped together into contours, according to the Gestalt laws of proximity, good continuation and, more recently, closure (Fig. 17), and symmetry; Figures 24-26 – the final projected edge is a combination of different detected edge segments). Thus, it would have been obvious, in view of Ruzon, Kim, and Papari, to configure Pirchheim’s system as claimed by performing an edge detection to generate an edge model representing the object in the real environment. The motivation is to enhance the visual representation of objects in the virtual augmented reality display.
Claim 3 adds into claim 2 “wherein the contrast is a color contrast or color variation” (Ruzon, Figure 4 - Ideal Step Edge). Thus, it would have been obvious, in view of Ruzon, Kim, and Papari, to configure Pirchheim’s system as claimed by performing an edge detection to generate an edge model based on the color contrast of the object’s surfaces. The motivation is to enhance the visual representation of objects in the virtual augmented reality display.
Claim 4 adds into claim 2 “a visual-inertial simultaneously location and mapping (SLAM) component and wherein generating the point of interest map associated with the physical environment is based at least in part on a six-degree of freedom (6DOF) pose output by the SLAM component” (Pirchheim, 2 SYSTEM OVERVIEW AND MAP REPRESENTATION - The system architecture of our hybrid approach follows the standard two-component design of optimization-based SLAM… The tracking component performs robust frame-rate map tracking of 6DOF or rotation-only camera motion and selects new keyframe candidates. In our system, keyframes can either be fully localized with a 6DOF pose, or panorama keyframes that are described with a rotation relative to a reference pose).
Claim 5 adds into claim 4 “wherein locating the line in the virtual scene representative of the physical environment further comprises: parameterizing the line using a first endpoint; placing the line using pairs of image associated with the image data, the first endpoint and the 6DOF pose” (Pirchheim, 2 SYSTEM OVERVIEW AND MAP REPRESENTATION - The map is represented as a collection of keyframes that store the camera pose Ci of the keyframe and an image pyramid for tracking points and finding new correspondences; Figure 2: Relationships in our hybrid map representation between keyframes and features depicted in two stages. In stage (a) 6DOF keyframes observe finite map features... In stage (b) infinite features are triangulated from corresponding observations matched between additional 6DOF keyframes and/or localized panorama keyframes from different local panorama maps; 4.1 6DOF mapping - 6DOF keyframes can be selected whenever we have a camera pose that is fully constrained in 6DOF. This is the case if enough finite feature points are part of the pose estimation as described in Section 3.1. Furthermore, we select regular 6DOF keyframes when they generate enough parallax to existing keyframes while imaging a new part of the scene. Parallax is required for robust feature triangulation).
Claim 6 adds into claim 2 “wherein forming the line associated with the physical environment is based at least in part on a first descriptor associated with the first line segment and a second descriptor associated with the second line segment” (Ruzon, Figure 6) (Noted: the example in figure 6 shows three (3) distinct regions of different descriptors (e.g., an edge segment formed by black and white regions; another edge segment formed by white and blue regions). Thus, it would have been obvious, in view of Ruzon, Kim, and Papari, to configure Pirchheim’s system as claimed by performing an edge detection of the object associated to edge segments of different descriptors. The motivation is to enhance the visual representation of objects in the virtual augmented reality display.
Claim 7 adds into claim 6 “wherein the first descriptor represents a first dominate color adjacent to the first line segment and the second descriptor represents a second dominate color adjacent to the second line segment” (Ruzon, Figure 6) (Noted: the example in figure 6 shows three (3) distinct regions of different descriptors (e.g., an edge segment formed by black and white regions where black is the first dominate color; another edge segment formed by white and blue regions where blue is the second dominate color). Thus, it would have been obvious, in view of Ruzon, Kim, and Papari, to configure Pirchheim’s system as claimed by performing an edge detection of the object associated to edge segments of different dominate colors. The motivation is to enhance the visual representation of objects in the virtual augmented reality display.
Claim 8 adds into claim 2 “adjusting the line within the virtual scene using a non-linear least squares model” which is well known in the art as “a non-linear least squares model” is a standard and highly effective approach for fitting curves to discrete edge feature samples in computer vision and image processing.
Claim 9 adds into claim 2 “locating the line in a model representative of the physical environment prior to projecting the line into the virtual scene” which is obvious because in an augmented reality AR space, the physical environment is pre-existed, then the detected edges of its objects can be early detected and projected into the virtual space scene allowing virtual objects interacting with the real world, like a virtual ball bouncing off your physical wall.
Claims 10-16 claim 1 method based on the system of claims 2-9; therefore, they are rejected under a similar rationale.
As per claim 17, Pirchheim teaches the claimed “method” comprising: “receiving image data associated with a physical environment” (Pirchheim, Figure 1: Rotation-only camera movements are handled by tracking and mapping local panorama maps registered within a global 3D map. The information contained in the panorama maps is also used for 3D reconstruction). It is noted that Pirchheim does not teach, but Ruzon teaches “generating, based at least in part on the image data, a plurality of edgelets associated with the physical environment” (Ruzon, 4 EDGE DETECTION - Here, we present a method of estimating the strength and orientation of an edge hypothesized to split a window. To summarize, we divide the window in half with a line segment, compute a color signature for each half, and find the EMD between them. Repeating this process using line segments with different orientations reveals the true strength and orientation of the edge; 5 JUNCTION DETECTION - Fig. 6 shows different line segments representing edges); “joining adjacent edgelets of the plurality of edgelets to generate a plurality of joined edgelets” (Ruzon, 5 JUNCTION DETECTION and 6 CORNER DETECTION – the detected junction or corners which are used to combine different detected edge segments; see also Kim, Figures 3E-3F - FIG. 3E shows a linked edge line formed by connecting edge point A and B according to an embodiment of the invention. The linked edge line has a slope S and the edge points A and B have directions DA and DB respectively. The linked edge line is formed due to DA and DB compatibility. However, if angle difference of slope S to DA or DB is significant, inconsistent data is generated in the missing block); “estimating, based at least in part on the plurality of joined edgelets, at least one contour associated with the physical environment” (Ruzon, 7.1 Edges and Junctions, Figures 13-14 – contours of the objects in the environment); “determining, based at least in part on the at least one contour, a surface” (Papari, 4.2. Grouping pixels into contours according to Gestalt principles, Figure 17 - Edge pixels can be grouped together into contours, according to the Gestalt laws of proximity, good continuation and, more recently, closure (Fig. 17), and symmetry; Figures 24-26 – the contour is a combination of different detected edge segments representing the surfaces of closed edge segments); and “projecting the surface into a virtual scene of the physical environment” (Ruzon, 7.1 Edges and Junctions - Fig. 11 contains four regions, two light and two dark. The edge separating the two dark regions has relatively low contrast; see also Papari, 4.2. Grouping pixels into contours according to Gestalt principles, Figure 17 - Edge pixels can be grouped together into contours, according to the Gestalt laws of proximity, good continuation and, more recently, closure (Fig. 17), and symmetry; Figures 24-26 – the final projected edge is a combination of different detected edge segments). Thus, it would have been obvious, in view of Ruzon, Kim, and Papari, to configure Pirchheim’s system as claimed by performing an edge detection to generate an edge model representing the object in the real environment. The motivation is to enhance the visual representation of objects in the virtual augmented reality display.
Claims 18 adds into claim 17 “wherein each of the plurality of edgelets define a small patch of the image data having an image gradient greater than a threshold” (Papari, Fig. 28. R–P plot related to the Canny edge detector for different values of the higher threshold tH; Kim, claim 2 - wherein an edge point is detected when the adjacent pixel difference exceeds a threshold).
Claims 19 adds into claim 17 “wherein a first edgelet of the plurality of edgelets is joined to a second edgelet of the plurality of edgelets in response to the first edgelet being less than or equal to a pixel threshold from the second edgelet” which is obvious because given any two (2) edgelets with their corresponding pixel thresholds, one of the pixel threshold must be less than or equal to other pixel threshold.
Claims 20 adds into claim 17 “wherein estimating the at least one contour associated with the physical environment further comprises: computing a reprojection error for each joined edgelet associated with the at least one contour; and rejecting a joined edgelet when the reprojection error is greater than or equal to a reprojection error threshold” which is obvious due to a “high” error in line projection because the reprojection error, by definition, is the pixel distance between an observed feature in an image and the assumed position predicted by a 3D model and camera pose (i.e., the geometric Euclidean distance between a real detected point and where the camera model expects that point should be).
Claims 21 adds into claim 17 “wherein determining the surface further comprises applying a Poisson reconstruction technique to the at least one contour” which is well-known in the art since the Poisson Surface Reconstruction is a highly effective global optimization technique used to determine a continuous, watertight 3D surface from oriented data samples. When applying it to contours, the process translates 2D or 3D contour lines into a smooth, closed volumetric boundary.
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 2-21 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 11481024. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the US patent, in view of the Papari, Pirchheim, Ruzon, and Kim reference, contains, or makes obvious, all features of the claims of the application.
Claims of this application
Claims of the US patent
2. A system comprising: one or more sensors for generating image data associated with a physical environment; a display; one or more processors; and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: generating, based at least in part on the image data, a point of interest map associated with the physical environment; identifying, based at least in part on a contrast associated with a portion of the image data, at least a first line segment and a second line segment associated with the physical environment; forming a line associated with the physical environment based at least in part on the first line segment and the second line segment; and projecting the line into a virtual scene representative of the physical environment.
2. A method comprising: presenting, on a display of a virtual reality or mixed reality system, a virtual scene to a user; detecting an edge of a physical object within a physical environment from one or more images captured by the virtual reality or mixed reality system, the physical object not displayed as part of the virtual scene; …
wherein detecting the edge of the physical object includes: detecting a first line segment based on a first color gradient within the one or more images; detecting a second line segment based on a second color gradient within the one or more images; merging the first line segment to the second line segment into the edge based at least in part on a similarly of the first color gradient to the second color gradient; and locating the edge within a three-dimensional model of the physical environment.
3. The system as recited in claim 2, wherein the contrast is a color contrast or color variation.
2… detecting a first line segment based on a first color gradient within the one or more images; detecting a second line segment based on a second color gradient within the one or more images…
4. The system as recited in claim 2, further comprising a visual-inertial simultaneously location and mapping (SLAM) component and wherein generating the point of interest map associated with the physical environment is based at least in part on a six-degree of freedom (6DOF) pose output by the SLAM component.
The Pirchheim reference (Pirchheim, 2 SYSTEM OVERVIEW AND MAP REPRESENTATION - The system architecture of our hybrid approach follows the standard two-component design of optimization-based SLAM… The tracking component performs robust frame-rate map tracking of 6DOF or rotation-only camera motion and selects new keyframe candidates. In our system, keyframes can either be fully localized with a 6DOF pose, or panorama keyframes that are described with a rotation relative to a reference pose).
5. The system as recited in claim 4, wherein locating the line in the virtual scene representative of the physical environment further comprises: parameterizing the line using a first endpoint; placing the line using pairs of image associated with the image data, the first endpoint and the 6DOF pose.
The Pirchheim reference (Pirchheim, 2 SYSTEM OVERVIEW AND MAP REPRESENTATION - The map is represented as a collection of keyframes that store the camera pose Ci of the keyframe and an image pyramid for tracking points and finding new correspondences; Figure 2: Relationships in our hybrid map representation between keyframes and features depicted in two stages. In stage (a) 6DOF keyframes observe finite map features... In stage (b) infinite features are triangulated from corresponding observations matched between additional 6DOF keyframes and/or localized panorama keyframes from different local panorama maps; 4.1 6DOF mapping - 6DOF keyframes can be selected whenever we have a camera pose that is fully constrained in 6DOF. This is the case if enough finite feature points are part of the pose estimation as described in Section 3.1. Furthermore, we select regular 6DOF keyframes when they generate enough parallax to existing keyframes while imaging a new part of the scene. Parallax is required for robust feature triangulation).
6. The system as recited in claim 2, wherein forming the line associated with the physical environment is based at least in part on a first descriptor associated with the first line segment and a second descriptor associated with the second line segment.
2. The method as recited in claim 1, wherein detecting the edge of the physical object includes: detecting a first line segment based on a first color gradient within the one or more images; detecting a second line segment based on a second color gradient within the one or more images; merging the first line segment to the second line segment into the edge based at least in part on a similarly of the first color gradient to the second color gradient; and locating the edge within a three-dimensional model of the physical environment.
7. The system as recited in claim 6, wherein the first descriptor represents a first dominate color adjacent to the first line segment and the second descriptor represents a second dominate color adjacent to the second line segment.
2. The method as recited in claim 1, wherein detecting the edge of the physical object includes: detecting a first line segment based on a first color gradient within the one or more images; detecting a second line segment based on a second color gradient within the one or more images; merging the first line segment to the second line segment into the edge based at least in part on a similarly of the first color gradient to the second color gradient; and locating the edge within a three-dimensional model of the physical environment.
8. The system as recited in claim 2, further comprising adjusting the line within the virtual scene using a non-linear least squares model.
which is well known in the art as “a non-linear least squares model” is a standard and highly effective approach for fitting curves to discrete edge feature samples in computer vision and image processing.
9. The system as recited in claim 2, further comprising locating the line in a model representative of the physical environment prior to projecting the line into the virtual scene.
8. The method as recited in claim 7, wherein the indicator is at least one of a ray projecting from the edge or the edge into a field of view of the user.
Claims 10-16 claim 1 method based on the system of claims 2-9; therefore, they are rejected under a similar rationale.
17. A method comprising:receiving image data associated with a physical environment; generating, based at least in part on the image data, a plurality of edgelets associated with the physical environment; joining adjacent edgelets of the plurality of edgelets to generate a plurality of joined edgelets; estimating, based at least in part on the plurality of joined edgelets, at least one contour associated with the physical environment; determining, based at least in part on the at least one contour, a surface; and projecting the surface into a virtual scene of the physical environment.
2. A method comprising: presenting, on a display of a virtual reality or mixed reality system, a virtual scene to a user; detecting an edge of a physical object within a physical environment from one or more images captured by the virtual reality or mixed reality system, the physical object not displayed as part of the virtual scene; …
wherein detecting the edge of the physical object includes: detecting a first line segment based on a first color gradient within the one or more images; detecting a second line segment based on a second color gradient within the one or more images; merging the first line segment to the second line segment into the edge based at least in part on a similarly of the first color gradient to the second color gradient; and locating the edge within a three-dimensional model of the physical environment.
18. The method as recited in claim 17, wherein each of the plurality of edgelets defines a small patch of the image data having an image gradient greater than a threshold.
Papari, Fig. 28. R–P plot related to the Canny edge detector for different values of the higher threshold tH; Kim, claim 2 - wherein an edge point is detected when the adjacent pixel difference exceeds a threshold
19. The method as recited in claim 17, wherein a first edgelet of the plurality of edgelets is joined to a second edgelet of the plurality of edgelets in response to the first edgelet being less than or equal to a pixel threshold from the second edgelet.
which is obvious because given any two (2) edgelets with their corresponding pixel thresholds, one of the pixel threshold must be less than or equal to other pixel threshold.
20. The method as recited in claim 17, wherein estimating the at least one contour associated with the physical environment further comprises: computing a reprojection error for each joined edgelet associated with the at least one contour; and rejecting a joined edgelet when the reprojection error is greater than or equal to a reprojection error threshold.
which is obvious due to a “high” error in line projection because the reprojection error, by definition, is the pixel distance between an observed feature in an image and the assumed position predicted by a 3D model and camera pose (i.e., the geometric Euclidean distance between a real detected point and where the camera model expects that point should be).
21. The method as recited in claim 17, wherein determining the surface further comprises applying a Poisson reconstruction technique to the at least one contour.
which is well-known in the art since the Poisson Surface Reconstruction is a highly effective global optimization technique used to determine a continuous, watertight 3D surface from oriented data samples. When applying it to contours, the process translates 2D or 3D contour lines into a smooth, closed volumetric boundary.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHU K NGUYEN whose telephone number is (571)272-7645. The examiner can normally be reached M-F 8-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel F. Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHU K NGUYEN/Primary Examiner, Art Unit 2616
Prosecution Insights