DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Information Disclosure Statement
2. The information disclosure statements (IDS) submitted on the following dates are in compliance with the provisions of 37 CFR 1.97 and are being considered by the Examiner: 03/28/2025; 07/28/2026.
Claim Rejections - 35 USC § 103
3. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
4. Claims 1, 4, 10, 13-14 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Frueh et al., (“Frueh”) [US-2018/0101984-A1] in view of Zhang et al. (machine translation of CN-107329593-A with citation below, hereinafter “Zhang”)
Regarding claim 1, Frueh discloses a calibration information determining method (Frueh- Fig. 17 and ¶0115, at least disclose an arrangement 1700 that is used to perform automatic calibration between a camera 1705 and a pose of an HMD; Fig. 18 and ¶0119, at least disclose a method 1800 for calibrating a camera that is used to capture images of a user wearing an HMD in an end-to-end mixed reality system; ¶0124, at least discloses At block 1825, the processor determines a camera calibration that establishes a relationship between the HMD features detected in images captured by the camera and the HMD pose corresponding to the captured image. The processor determines the camera calibration based on the set of 2-D/3-D point pairs) comprises:
obtaining a plurality of pairs of sampling information of the target device , wherein different pairs of sampling information correspond to different poses of the target device, and each pair of sampling information comprises: two-dimensional detection data of the target device and three-dimensional detection data of the target device, wherein the two-dimensional detection data is obtained by an image acquisition device, and the three-dimensional detection data is obtained by a virtual reality device (Frueh- Fig. 17 and ¶0117-0118, at least disclose “The calibration is performed on the basis of distinguishing features of the HMD in the different orientations […] For example, the calibration can be performed on the basis of an Aruco marker 1745 that is affixed to a surface of the HMD […] the calibration can be performed on the basis of a logo or other specific geometric features of the HMD such as edges, corners, cable connectors, and the like […] The camera 1705 captures images […] of the HMD in the corresponding orientations […] For each feature detected in the captured images […] the processor 1708 is configured to determine a 3-D location of the feature in the HMD coordinate system using a pose of the HMD for the corresponding orientations […] and the measured offsets. As discussed herein, the processor 1708 is able to acquire the pose information for the HMD in a telemetry stream received from the HMD. The processor 1708 is also able to identify pixel locations of the features in the captured images […] The 3-D location and pixel locations of the feature form a 2-D/3-D point pair. The processor 1708 is therefore able to generate a set of 2-D/3-D point pairs from the captured images […] and the corresponding pose information. The set of 2-D/3-D point pairs is then used to determine a projection matrix that projects the 3-D features onto the 2-D image plane of the camera 1705; ¶0121, at least discloses “At block 1810, the processor determines 3-D locations of the HMD features in the captured image in HMD coordinates. The 3-D locations of the HMD features can be determined by computing a central point (or other reference point) of the HMD using pose information in a telemetry stream provided to the processor by the HMD” Regarding what this "position information in a telemetry stream provided by the HMD" and whether it comprises the claimed "acquiring ... a pose of a target device": [0027] describes that the HMD can be such as an an Oculus Rift or HTC Vive provides 6Degree of Freedom (6DoF) pose data posture. And [0131] notes that, "The sensors 940, 945 in the electronic device 2005 are able to generate information indicating the six degree-of-freedom (6DoF) pose of the electronic device 2005, which includes a three-dimensional position of the electronic device 2005 and a three-dimensional orientation of the electronic device 2005”); and
determining calibration information of the image acquisition device in a coordinate system of a virtual reality world based on the plurality of pairs of sampling information calibrated based on 2D / 3D point pairs determined for an HMD taken in exemplary orientations (poses) 1710, 1715 and 1720. With respect to determining calibration image of the camera "according to ... the pose of the target device," as discussed above with respect to [0131], the HMD data stream used to determine 3D HMD feature coordinates includes 3DoF position and 3DoF orientation (pose); ¶0120-0122, at least disclose “At block 1805, the processor identifies 2-D pixel locations of one or more HMD features in a captured image that includes the HMD in a first location and orientation […] At block 1810, the processor determines 3-D locations of the HMD features in the captured image in HMD coordinates. The 3-D locations of the HMD features can be determined by computing a central point (or other reference point) of the HMD using pose information in a telemetry stream provided to the processor by the HMD […] At block 1815, the processor stores a 2-D/3-D point pair that includes information indicating the 2-D location of each pixel associated with the HMD feature and a corresponding 3-D location of the portion of the HMD feature represented by the pixel”; ¶0124, at least discloses “At block 1825, the processor determines a camera calibration that establishes a relationship between the HMD features detected in images captured by the camera and the HMD pose corresponding to the captured image. The processor determines the camera calibration based on the set of 2-D/3-D point pairs”).
Frueh does not explicitly disclose during a process of a movement along a plurality of dimensional directions of a target device, obtaining a plurality of pairs of sampling information of the target device corresponding to the plurality of dimensional directions; the plurality of dimensional directions.
However, Zhang discloses
during a process of a movement along a plurality of dimensional directions of a target device, obtaining a plurality of pairs of sampling information of the target device corresponding to the plurality of dimensional directions (Zhang- ¶0008-0035, at least disclose “Photographing a real scene including at least two VR handles to obtain a two-dimensional image, the at least two VR handles being respectively surrounded by rings with different visual characteristics […] Performing target recognition training on visual features of a ring surrounding each VR handle in the plurality of different-orientation two-dimensional images to establish a target recognition model capable of recognizing a visual feature of a ring surrounding each VR handle […] obtaining the three-dimensional coordinates of each VR handle in the real scene according to the shooting parameters and the point coordinates of the circle surrounding each VR handle in the two-dimensional image […] Photographing a real scene including the at least two VR handles from different directions to obtain a plurality of two-dimensional images with different orientations […] According to the distance between the real annulus of the first circle surrounding the first VR handle and the optical center and the three-dimensional coordinate point cluster of the first VR handle, a first The three-dimensional coordinates of the center of the ring in the real scene as the three-dimensional coordinates of the first VR handle in the real scene […] Map each VR handle to a virtual scene corresponding to the real scene according to the three-dimensional coordinates of each VR handle in the real scene and the holding end of each VR handle to obtain a VR image; The VR image is transmitted to the user's head-mounted device for display”; ¶0077, at least discloses “Specifically, a real scene including at least two VR handles is photographed from different directions to obtain multiple two-dimensional images with different orientations”; ¶0143, at least discloses “the position of the VR handle in the real scene may be acquired in real time. When the user moves the VR handle, the virtual engine may perform real-time mobile rendering of the virtual model in the rendering engine and transmit the result to the head-up for display” [Wingdings font/0xE0] suggests during a process of a movement along a plurality of dimensional directions of a target device, obtaining a plurality of pairs of sampling information of the target device corresponding to the plurality of dimensional directions).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Frueh to incorporate the teachings of Zhang, and apply obtaining sampling information of the VR hand controller in real time from different directions during the movement process into Frueh’s teachings in order during a process of a movement along a plurality of dimensional directions of a target device, obtaining a plurality of pairs of sampling information of the target device corresponding to the plurality of dimensional directions, wherein different pairs of sampling information correspond to different poses of the target device, and each pair of sampling information comprises: two-dimensional detection data of the target device and three-dimensional detection data of the target device, wherein the two-dimensional detection data is obtained by an image acquisition device, and the three-dimensional detection data is obtained by a virtual reality device; and determining calibration information of the image acquisition device in a coordinate system of a virtual reality world based on the plurality of pairs of sampling information corresponding to the plurality of dimensional directions.
Doing so would be able to obtain sampling information in multiple dimension directions and improve the calibration accuracy.
Regarding claim 4, Frueh in view of Zhang, discloses the method of claim 1, and further discloses wherein a sampling time parameter of the two-dimensional detection data is an image acquisition time when the image acquisition device obtains the two-dimensional detection data, and the sampling time parameter of the three-dimensional detection data is a detection time when the virtual reality device detects the three-dimensional detection data (Frueh- Fig. 17 and ¶0117-0118, at least disclose “The calibration is performed on the basis of distinguishing features of the HMD in the different orientations […] For example, the calibration can be performed on the basis of an Aruco marker 1745 that is affixed to a surface of the HMD […] the calibration can be performed on the basis of a logo or other specific geometric features of the HMD such as edges, corners, cable connectors, and the like […] The camera 1705 captures images […] of the HMD in the corresponding orientations […] For each feature detected in the captured images […] the processor 1708 is configured to determine a 3-D location of the feature in the HMD coordinate system using a pose of the HMD for the corresponding orientations […] and the measured offsets. As discussed herein, the processor 1708 is able to acquire the pose information for the HMD in a telemetry stream received from the HMD. The processor 1708 is also able to identify pixel locations of the features in the captured images […] The 3-D location and pixel locations of the feature form a 2-D/3-D point pair. The processor 1708 is therefore able to generate a set of 2-D/3-D point pairs from the captured images […] and the corresponding pose information. The set of 2-D/3-D point pairs is then used to determine a projection matrix that projects the 3-D features onto the 2-D image plane of the camera 1705; ¶0121, at least discloses “At block 1810, the processor determines 3-D locations of the HMD features in the captured image in HMD coordinates. The 3-D locations of the HMD features can be determined by computing a central point (or other reference point) of the HMD using pose information in a telemetry stream provided to the processor by the HMD” Regarding what this "position information in a telemetry stream provided by the HMD" and whether it comprises the claimed "acquiring ... a pose of a target device": [0027] describes that the HMD can be such as an an Oculus Rift or HTC Vive provides 6Degree of Freedom (6DoF) pose data posture. And [0131] notes that, "The sensors 940, 945 in the electronic device 2005 are able to generate information indicating the six degree-of-freedom (6DoF) pose of the electronic device 2005, which includes a three-dimensional position of the electronic device 2005 and a three-dimensional orientation of the electronic device 2005” [Wingdings font/0xE0] The corresponding 3D key points and 2D key points can correspond to the same feature points on the target device. Obtaining the sampling time parameter of the two-dimensional detection data as the image acquisition moment when the image acquisition device acquires the two dimensional detection data, and the sampling time parameter of the three-dimensional detection data as the detection moment when the virtual reality device detects the three-dimensional detection data according to the actual needs).
Regarding claim 10, Frueh discloses a calibration information determining method (Frueh- Fig. 17 and ¶0115, at least disclose an arrangement 1700 that is used to perform automatic calibration between a camera 1705 and a pose of an HMD; Fig. 18 and ¶0119, at least disclose a method 1800 for calibrating a camera that is used to capture images of a user wearing an HMD in an end-to-end mixed reality system; ¶0124, at least discloses At block 1825, the processor determines a camera calibration that establishes a relationship between the HMD features detected in images captured by the camera and the HMD pose corresponding to the captured image. The processor determines the camera calibration based on the set of 2-D/3-D point pairs) comprises:
information is used for guiding a target device to move along (Frueh- ¶0033, at least discloses the processor 110 can record images of the user 125 taken by the camera 130 while the user 125 is watching a target image displayed on the screen 120. The target image moves to different locations on the screen 120 and the user 125 is expected to follow the target image with their eyes. The camera 130 captures an image of the user 125 while the user 125 is looking at each of the different locations on the screen 120 indicated by the target image; ¶0041, at least discloses Prior to capturing an image of the user 305 during the second instance 325, the target image 330 is moved to a different point in the grid 335 that is associated with a different location on the screen 300.);
sending a plurality of three-dimensional detection data of the target device 3-D location of the feature in the HMD coordinate system using a pose of the HMD for the corresponding orientations […] and the measured offsets. As discussed herein, the processor 1708 is able to acquire the pose information for the HMD in a telemetry stream received from the HMD. The processor 1708 is also able to identify pixel locations of the features in the captured images 1750, 1755, 1760. The 3-D location and pixel locations of the feature form a 2-D/3-D point pair. The processor 1708 is therefore able to generate a set of 2-D/3-D point pairs from the captured images 1750, 1755, 1760 and the corresponding pose information; ¶0124-0125, at least disclose At block 1825, the processor determines a camera calibration that establishes a relationship between the HMD features detected in images captured by the camera and the HMD pose corresponding to the captured image. The processor determines the camera calibration based on the set of 2-D/3-D point pairs […] The camera calibration generated by the method 1800 can then be used to generate mixed reality images including a user wearing the HMD by merging one or more VR images with images captured by the camera and a synthesized representation of a portion of the user's face that is occluded by the HMD; ¶0126, at least discloses FIG. 19 illustrates variations of measured parameters associated with motion of an HMD and images of the moving HMD […] The telemetry stream is time stamped and the projected HMD position 1905 is plotted as a function of the HMD timing reference on the basis of the timestamps in the telemetry stream); and
obtaining a calibration information of an image acquisition device in a coordinate system of a virtual reality world (Frueh- Fig. 17 and ¶0117-0118, at least disclose that the camera is calibrated based on 2D / 3D point pairs determined for an HMD taken in exemplary orientations (poses) 1710, 1715 and 1720. With respect to determining calibration image of the camera "according to ... the pose of the target device," as discussed above with respect to [0131], the HMD data stream used to determine 3D HMD feature coordinates includes 3DoF position and 3DoF orientation (pose); ¶0120-0122, at least disclose “At block 1805, the processor identifies 2-D pixel locations of one or more HMD features in a captured image that includes the HMD in a first location and orientation […] At block 1810, the processor determines 3-D locations of the HMD features in the captured image in HMD coordinates. The 3-D locations of the HMD features can be determined by computing a central point (or other reference point) of the HMD using pose information in a telemetry stream provided to the processor by the HMD […] At block 1815, the processor stores a 2-D/3-D point pair that includes information indicating the 2-D location of each pixel associated with the HMD feature and a corresponding 3-D location of the portion of the HMD feature represented by the pixel”; ¶0124, at least discloses “At block 1825, the processor determines a camera calibration that establishes a relationship between the HMD features detected in images captured by the camera and the HMD pose corresponding to the captured image. The processor determines the camera calibration based on the set of 2-D/3-D point pairs”).
Frueh does not explicitly disclose responding to a calibration starting operation, presenting operation prompt information, wherein the operation prompt information is used for guiding a target device to move along a plurality of dimensional directions; during a process of controlling the target device to move along the plurality of dimensional directions, sending a plurality of three-dimensional detection data of the target device corresponding to the plurality of dimensional directions; and responding to a sampling termination message, obtaining a calibration information of an image acquisition device in a coordinate system of a virtual reality world.
However, Zhang discloses
responding to a calibration starting operation, presenting operation prompt information, wherein the operation prompt information is used for guiding a target device to move along a plurality of dimensional directions (Zhang- ¶0008-0035, at least disclose “Photographing a real scene including at least two VR handles to obtain a two-dimensional image, the at least two VR handles being respectively surrounded by rings with different visual characteristics […] Performing target recognition training on visual features of a ring surrounding each VR handle in the plurality of different-orientation two-dimensional images to establish a target recognition model capable of recognizing a visual feature of a ring surrounding each VR handle […] obtaining the three-dimensional coordinates of each VR handle in the real scene according to the shooting parameters and the point coordinates of the circle surrounding each VR handle in the two-dimensional image […] Photographing a real scene including the at least two VR handles from different directions to obtain a plurality of two-dimensional images with different orientations […] According to the distance between the real annulus of the first circle surrounding the first VR handle and the optical center and the three-dimensional coordinate point cluster of the first VR handle, a first The three-dimensional coordinates of the center of the ring in the real scene as the three-dimensional coordinates of the first VR handle in the real scene […] Map each VR handle to a virtual scene corresponding to the real scene according to the three-dimensional coordinates of each VR handle in the real scene and the holding end of each VR handle to obtain a VR image; The VR image is transmitted to the user's head-mounted device for display”; ¶0077, at least discloses “Specifically, a real scene including at least two VR handles is photographed from different directions to obtain multiple two-dimensional images with different orientations”; ¶0143, at least discloses “the position of the VR handle in the real scene may be acquired in real time. When the user moves the VR handle, the virtual engine may perform real-time mobile rendering of the virtual model in the rendering engine and transmit the result to the head-up for display” [Wingdings font/0xE0] suggests during a process of a movement along a plurality of dimensional directions of a target device, obtaining a plurality of pairs of sampling information of the target device corresponding to the plurality of dimensional directions).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Frueh to incorporate the teachings of Zhang, and apply obtaining sampling information of the VR hand controller in real time from different directions during the movement process into Frueh’s teachings for responding to a calibration starting operation, presenting operation prompt information, wherein the operation prompt information is used for guiding a target device to move along a plurality of dimensional directions; during a process of controlling the target device to move along the plurality of dimensional directions, sending a plurality of three-dimensional detection data of the target device corresponding to the plurality of dimensional directions; and responding to a sampling termination message, obtaining a calibration information of an image acquisition device in a coordinate system of a virtual reality world.
Doing so would be able to obtain sampling information in multiple dimension directions and improve the calibration accuracy.
Regarding claim 13, Frueh discloses an electronic device (Frueh- Fig. 9 and ¶0068, at least disclose a display system 900 that includes an electronic device 905 configured to provide VR, AR, or MR functionality via a display […] the electronic device 905 can include a portable user device, such as an HMD, a tablet computer, computing-enabled cellular phone (e.g., a “smartphone”), a notebook computer, a personal digital assistant (PDA), a gaming console system, and the like. In other embodiments, the electronic device 905 can include a fixture device, such as medical imaging equipment, a security imaging sensor system, an industrial robot control system, a drone control system, and the like; Fig. 20 and ¶0129, at least disclose a display system 2000 that includes an electronic device 2005 configured to provide VR, AR, or MR functionality via a display), comprising:
a processor (Frueh- Figs. 16, 20 and ¶0130-0131, at least disclose The eye gaze stream can be provided in real time to other devices such as the processor 1620 shown in FIG. 16 […] The telemetry stream can be provided in real time to other devices such as the processor 1620 shown in FIG. 16); and,
a memory storing a program (Frueh- Fig. 16 and ¶0109, at least disclose The processor 1620 is configured to execute instructions, such as instructions stored in the memory 1625 and store the results of the instructions in the memory 1625. The processor 1620 can be implemented as an individual processor or as a distributed set of processors […] The processor 1620 can store the received information in the memory 1625);
wherein the program comprises instructions which, when executed by the processor, cause the processor to carry out a calibration information determining method (Frueh- Fig. 16 and ¶0109, at least disclose The processor 1620 is configured to execute instructions, such as instructions stored in the memory 1625 and store the results of the instructions in the memory 1625. The processor 1620 can be implemented as an individual processor or as a distributed set of processors […] The processor 1620 can store the received information in the memory 1625) comprises the method of claim 1.
Regarding claim 14, Frueh in view of Zhang, discloses a non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium stores computer instructions that, when executed by a processor of a computer (Frueh- Fig. 16 and ¶0109, at least disclose The processor 1620 is configured to execute instructions, such as instructions stored in the memory 1625 and store the results of the instructions in the memory 1625. The processor 1620 can be implemented as an individual processor or as a distributed set of processors […] The processor 1620 can store the received information in the memory 1625), cause the processor to perform the method of claim 1.
Regarding claim 22, Frueh in view of Zhang, discloses a non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium stores computer instructions that, when executed by a processor of a computer (Frueh- Fig. 16 and ¶0109, at least disclose The processor 1620 is configured to execute instructions, such as instructions stored in the memory 1625 and store the results of the instructions in the memory 1625. The processor 1620 can be implemented as an individual processor or as a distributed set of processors […] The processor 1620 can store the received information in the memory 1625), cause the processor to perform the method of claim 10.
5. Claims 2-3 and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Frueh in view of Zhang, further in view of Yi et al. (machine translation of WO-2020149462-A1 with citation below, hereinafter “Yi”)
Regarding claim 2, Frueh in view of Zhang, discloses the method of claim 1, and further discloses wherein the obtaining the plurality of pairs of sampling information of the target device corresponding to the plurality of dimensional directions (see Claim 1 rejection for detailed analysis), and the prior art does not explicitly disclose, but Yi discloses the method comprises:
determining a first time correction parameter based on a sampling time parameter of the two-dimensional detection data and a sampling time parameter of the three-dimensional detection data of the target device corresponding to a same pose (Yi- ¶0019, at least discloses “the error correction unit 2D coordinate detection unit for each camera to detect the 2D coordinate value of the motion image for each object detected by the detection unit; First three-dimensional coordinates (x1, y1) and second two-dimensional coordinates (x2, y2) respectively detected by the two-dimensional coordinate detection unit to which a position variable z matching the same position is added ( a 3D coordinate detector for detecting x1 ,y1,z1) and second 3D coordinates (x1 ,y1,z2); The spatial error coordinates (Δx, Δy, Δz) are calculated through the difference between the first 3D coordinates (x1 ,y1,z1) and the second 3D coordinates (x2,y2,z2) detected by the 3D coordinate detector […] Time to calculate time error coordinates (Δx, Δy, Δz) through the difference between the third 3D coordinates (x1,y1,t1) and the fourth 3D coordinates (x2,y2,t1) synchronized in time in the time synchronization. An error calculation unit; In correspondence with the corrected 3D coordinates (x1', y1', t1) generated by the time error calculation unit and the same time (t1'), the first 2D coordinates (x1, y1) respectively detected by the 2D coordinate detection unit )”); and
aligning the sampling time parameter of a plurality of the two-dimensional detection data and the sampling time parameter of a plurality of the three-dimensional detection data based on the first time correction parameter (Yi- ¶0019, at least discloses “First two-dimensional coordinates respectively detected by the two-dimensional coordinate detection unit by performing time synchronization based on the calculated spatial error coordinates using the spatial error coordinates (Δx, Δy, Δz) calculated by the spatial error calculation unit. By applying a time variable t for time synchronization to (x1,y1) and second 2D coordinates (x2,y2), the third 3D coordinates (x1,y1,t1) and the fourth 3D coordinates (x2, a time synchronization unit detecting y2,t1 ); Time to calculate time error coordinates (Δx, Δy, Δz) through the difference between the third 3D coordinates (x1,y1,t1) and the fourth 3D coordinates (x2,y2,t1) synchronized in time in the time synchronization. An error calculation unit; In correspondence with the corrected 3D coordinates (x1', y1', t1) generated by the time error calculation unit and the same time (t1'), the first 2D coordinates (x1, y1) respectively detected by the 2D coordinate detection unit ) And the fifth 3D coordinates (x1,y1,t1') and the 6th 3D coordinates (x2,y2,t1') of the second 2D coordinates (x2,y2), respectively, and the corrected 3D coordinates 2nd order in the object image through the difference between the coordinates (x1',y1',t1) and the detected fifth 3D coordinates (x1,y1,t1') and the 6th 3D coordinates (x2,y2,t1') A three-dimensional coordinate error detection unit that calculates a coordinate error (Δx, Δy); And a two-dimensional coordinate error detection unit that repeats the error of the two-dimensional coordinates until the calculated error (Δx, Δy) of the calculated two-dimensional coordinates is greater than a preset value, transmitted to the detection unit and converges to a preset value”).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Frueh/Zhang to incorporate the teachings of Yi, and apply time synchronization into Frueh/Zhang’s teachings for determining a first time correction parameter based on a sampling time parameter of the two-dimensional detection data and a sampling time parameter of the three-dimensional detection data of the target device corresponding to a same pose; and aligning the sampling time parameter of a plurality of the two-dimensional detection data and the sampling time parameter of a plurality of the three-dimensional detection data based on the first time correction parameter.
Doing so would improves accuracy below the resolution of a camera by repeatedly applying correction of temporal and spatial errors in calculating 3D coordinates of a moving object acquired and tracked by a plurality of cameras.
Regarding claim 3, Frueh in view of Zhang, discloses the method of claim 1, and further discloses wherein the obtaining the plurality of pairs of sampling information of the target device corresponding to the plurality of dimensional directions (see Claim 1 rejection for detailed analysis) comprises:
a plurality of three-dimensional detection data of the target device corresponding to the plurality of dimensional directions (Zhang- ¶0008-0035, at least disclose “Photographing a real scene including at least two VR handles to obtain a two-dimensional image, the at least two VR handles being respectively surrounded by rings with different visual characteristics […] Performing target recognition training on visual features of a ring surrounding each VR handle in the plurality of different-orientation two-dimensional images to establish a target recognition model capable of recognizing a visual feature of a ring surrounding each VR handle […] obtaining the three-dimensional coordinates of each VR handle in the real scene according to the shooting parameters and the point coordinates of the circle surrounding each VR handle in the two-dimensional image […] Photographing a real scene including the at least two VR handles from different directions to obtain a plurality of two-dimensional images with different orientations […] According to the distance between the real annulus of the first circle surrounding the first VR handle and the optical center and the three-dimensional coordinate point cluster of the first VR handle, a first The three-dimensional coordinates of the center of the ring in the real scene as the three-dimensional coordinates of the first VR handle in the real scene […] Map each VR handle to a virtual scene corresponding to the real scene according to the three-dimensional coordinates of each VR handle in the real scene and the holding end of each VR handle to obtain a VR image; The VR image is transmitted to the user's head-mounted device for display”; ¶0077, at least discloses “Specifically, a real scene including at least two VR handles is photographed from different directions to obtain multiple two-dimensional images with different orientations”; ¶0143, at least discloses “the position of the VR handle in the real scene may be acquired in real time. When the user moves the VR handle, the virtual engine may perform real-time mobile rendering of the virtual model in the rendering engine and transmit the result to the head-up for display”).
The prior art does not explicitly disclose determining the three-dimensional detection data which is aligned with a sampling time of the two-dimensional detection data based on at least two three-dimensional detection data in a plurality of three-dimensional detection data of the target device corresponding to the plurality of dimensional directions.
However, Yi discloses
determining the three-dimensional detection data which is aligned with a sampling time of the two-dimensional detection data based on at least two three-dimensional detection data (Yi- ¶0019, ¶0064-0079, at least disclose a time synchronization unit, which uses the spatial error coordinates (Δx, Δy, Δz) calculated by the spatial error calculation unit to perform time synchronization based on the calculated spatial error coordinates, applies a time variable (t) for time synchronization to the first two-dimensional coordinate (x1, y1) and the second two-dimensional coordinate (x2, y2) respectively detected by the two-dimensional coordinate detection unit, and detects the third three-dimensional coordinate (x1, y1, t1) and the fourth three-dimensional coordinate (x2, y2, t1); a time error calculation unit, which calculates the time error coordinates (Δx, Δy, Δt) based on the difference between the third three-dimensional coordinate (x1, y1, t1) and the fourth three-dimensional coordinate (x2, y2, t1) synchronized in time; a three-dimensional coordinate error detection unit, which, corresponding to the corrected three-dimensional coordinate (x1', y1', t1) generated by the time error calculation unit, detects the fifth three-dimensional coordinate (x1, y1, t1') and the sixth three-dimensional coordinate (x2, y2, t1') of the first two-dimensional coordinate (xl, yl) and the second two-dimensional coordinate (x2, y2) respectively detected by the two-dimensional coordinate detection unit at the same time (tl'), and calculates the error (Δx, Δy) of the second coordinate in the object image based on the difference between the corrected three-dimensional coordinate (x1', y1', t1) and the detected fifth three-dimensional coordinate (x1, y1, t1') and the sixth three-dimensional coordinate (x2, y2, t1'); and a two-dimensional coordinate error detection unit, which, when the range of the calculated two-dimensional coordinate error (Δx, Δy) is greater than a predetermined value, sends the calculated two-dimensional coordinate error (fix, fiy) to the detection unit and repeats the calculation of the two-dimensional coordinate error until the calculated two-dimensional coordinate error (Δx, Δy) converges to a predetermined value).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Frueh/Zhang to incorporate the teachings of Yi, and apply time synchronization into Frueh/Zhang’s teachings for determining the three-dimensional detection data which is aligned with a sampling time of the two-dimensional detection data based on at least two three-dimensional detection data in a plurality of three-dimensional detection data of the target device corresponding to the plurality of dimensional directions.
Doing so would perform correction and alignment on multiple sets of three-dimensional detection data.
The system of claims 15-16 are similar in scope to the functions performed by the method of claims 2-3 and therefore claims 15-16 are rejected under the same rationale.
6. Claims 5 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Frueh in view of Zhang, further in view of Huang et al. (machine translation of WO-2021082801-A1 with citation below, hereinafter “Huang”)
Regarding claim 5, Frueh in view of Zhang, discloses the method of claim 1, and further discloses wherein the method further comprises:
determining that a plurality of pairs of sampling information of the target device in the plurality of the dimensional directions (see Claim 1 rejection for detailed analysis)
a distribution of the plurality of pairs of sampling information in different dimensional directions meets a preset distribution uniformity (Frueh- ¶0141, at least discloses The visible surfaces of the HMD are usually a uniform color so that the location of the Aruco marker can be inpainted with a fixed color that is computed as an average of the non-marker pixels surrounding the marker area on the surface of the HMD); and
the target device completes the movement along the plurality of the dimensional
directions (see Claim 1 rejection for detailed analysis).
The prior art does not explicitly disclose determining that a plurality of pairs of sampling information of the target device in the plurality of the dimensional directions meet a sampling termination condition, wherein the sampling termination condition comprises at least one of the following conditions: a number of the plurality of pairs of sampling information is larger than a preset number; a distribution of the plurality of pairs of sampling information in different dimensional directions meets a preset distribution uniformity.
However, Huang discloses
determining that a plurality of pairs of sampling information of the target device in the plurality of the dimensional directions meet a sampling termination condition (Huang- page 9, 1st paragraph, at least discloses a certain number of matching pairs ( for example, 7 pairs, 8 pairs, etc.) are randomly selected from the matching pairs between the two-dimensional feature points of the current frame image and the two-dimensional feature points of the reference image, and the selected matching pairs are calculated The basic matrix or essential matrix between the current frame image and the reference image is based on the epipolar constraint. If a two-dimensional feature point is far away from the corresponding epipolar line, for example, greater than a threshold, the two-dimensional feature can be considered. The points are mismatched points. Through the random sampling process of a certain number of iterations, the random sampling result with the largest number of internal points is selected as the final matching result), wherein the sampling termination condition comprises at least one of the following conditions:
a number of the plurality of pairs of sampling information is larger than a preset number (Huang- page 9, 1st paragraph, at least discloses a certain number of matching pairs ( for example, 7 pairs, 8 pairs, etc.) are randomly selected from the matching pairs between the two-dimensional feature points of the current frame image and the two-dimensional feature points of the reference image, and the selected matching pairs are calculated The basic matrix or essential matrix between the current frame image and the reference image is based on the epipolar constraint. If a two-dimensional feature point is far away from the corresponding epipolar line, for example, greater than a threshold, the two-dimensional feature can be considered The points are mismatched points. Through the random sampling process of a certain number of iterations, the random sampling result with the largest number of internal points is selected as the final matching result).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Frueh/Zhang to incorporate the teachings of Huang, and apply the points greater than a threshold into Frueh/Zhang’s teachings for determining that a plurality of pairs of sampling information of the target device in the plurality of the dimensional directions meet a sampling termination condition, wherein the sampling termination condition comprises at least one of the following conditions: a number of the plurality of pairs of sampling information is larger than a preset number.
Doing so would provide an augmented reality processing method.
The system of claim 18 is similar in scope to the functions performed by the method of claim 5 and therefore claim 18 is rejected under the same rationale.
7. Claims 6 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Frueh in view of Zhang, further in view of Li et al. (machine translation of CN-112000228-A with citation below, hereinafter “Li”)
Regarding claim 6, Frueh in view of Zhang, discloses the method of claim 1, and further discloses wherein a movement path along which the target device moves along the plurality of dimensional directions (see Claim 1 rejection for detailed analysis), and the prior art does not explicitly disclose, but Li discloses the method comprises a circular movement path and a linear movement path, a movement direction of the linear movement path intersecting a plane defined by the circular movement path (Li- Fig. 1 shows the direction of movement, the direction of rotation of the wheel and the normal direction of the movement surface supporting its movement when the wheel is rolling; ¶0039, at least discloses When the wheel is rolling, the trajectory of a certain mass point on the surface of the wheel is a perfect circle, which is a very special case of the "curve" mentioned in the present invention. In the present invention, the user's limbs are required to perform curvilinear movements. For example, the limbs can perform wheel rolling movements. Of course, the user's limbs are not required to perform circular curve movements, nor do they need to be a complete circle of curve movements, as long as there is a small arc. The curve movement of the curve (the curvature radius of the arc on the curve can be different) is enough, and the movement of the character in the virtual reality can be controlled by the curve movement of the limbs).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Frueh/Zhang to incorporate the teachings of Li, and apply the circular curve movements into Frueh/Zhang’s teachings in order a movement path along which the target device moves along the plurality of dimensional directions comprises a circular movement path and a linear movement path, a movement direction of the linear movement path intersecting a plane defined by the circular movement path.
Doing so would control movement in immersive virtual reality.
The system of claim 19 is similar in scope to the functions performed by the method of claim 6 and therefore claim 19 is rejected under the same rationale.
8. Claims 7-8 and 20-21 are rejected under 35 U.S.C. 103 as being unpatentable over Frueh in view of Zhang, further in view of Liu et al. (machine translation of CN-113538591-A with citation below, hereinafter “Liu”)
Regarding claim 7, Frueh in view of Zhang, discloses the method of claim 1, and further discloses wherein the method further comprises:
performing screening of sampling information for the plurality of pairs of sampling information (Frueh- Fig. 17 and ¶0117-0118, at least disclose that the camera is calibrated based on 2D / 3D point pairs determined for an HMD taken in exemplary orientations (poses) 1710, 1715 and 1720. With respect to determining calibration image of the camera "according to ... the pose of the target device," as discussed above with respect to [0131], the HMD data stream used to determine 3D HMD feature coordinates includes 3DoF position and 3DoF orientation (pose); ¶0120-0122, at least disclose “At block 1805, the processor identifies 2-D pixel locations of one or more HMD features in a captured image that includes the HMD in a first location and orientation […] At block 1810, the processor determines 3-D locations of the HMD features in the captured image in HMD coordinates. The 3-D locations of the HMD features can be determined by computing a central point (or other reference point) of the HMD using pose information in a telemetry stream provided to the processor by the HMD […] At block 1815, the processor stores a 2-D/3-D point pair that includes information indicating the 2-D location of each pixel associated with the HMD feature and a corresponding 3-D location of the portion of the HMD feature represented by the pixel”; ¶0124, at least discloses “At block 1825, the processor determines a camera calibration that establishes a relationship between the HMD features detected in images captured by the camera and the HMD pose corresponding to the captured image. The processor determines the camera calibration based on the set of 2-D/3-D point pairs”));
aligning the plurality of pairs of sampling information ; and
performing screening of sampling information for the plurality of pairs of sampling information (Frueh- Fig. 17 and ¶0117-0118, at least disclose that the camera is calibrated based on 2D / 3D point pairs determined for an HMD taken in exemplary orientations (poses) 1710, 1715 and 1720. With respect to determining calibration image of the camera "according to ... the pose of the target device," as discussed above with respect to [0131], the HMD data stream used to determine 3D HMD feature coordinates includes 3DoF position and 3DoF orientation (pose); ¶0120-0122, at least disclose “At block 1805, the processor identifies 2-D pixel locations of one or more HMD features in a captured image that includes the HMD in a first location and orientation […] At block 1810, the processor determines 3-D locations of the HMD features in the captured image in HMD coordinates. The 3-D locations of the HMD features can be determined by computing a central point (or other reference point) of the HMD using pose information in a telemetry stream provided to the processor by the HMD […] At block 1815, the processor stores a 2-D/3-D point pair that includes information indicating the 2-D location of each pixel associated with the HMD feature and a corresponding 3-D location of the portion of the HMD feature represented by the pixel”; ¶0124, at least discloses “At block 1825, the processor determines a camera calibration that establishes a relationship between the HMD features detected in images captured by the camera and the HMD pose corresponding to the captured image. The processor determines the camera calibration based on the set of 2-D/3-D point pairs”)).
The prior art does not explicitly disclose a first screening parameter, wherein the first screening parameter is a first projection error threshold; on a second time correction parameter; a second screening parameter, wherein the second screening parameter is a second projection error threshold.
However, Liu discloses
a first screening parameter, wherein the first screening parameter is a first projection error threshold (Liu- ¶0018-0020, at least disclose “Construct a reprojection error function model; Iterative calculation is performed according to the target point pair and the reprojection error function model to obtain the minimum reprojection error and the external parameters between the distance measuring device and the camera. Further, after the iterative calculation is performed according to the target point pair and the reprojection error function model to obtain the minimum reprojection error and the external parameters between the distance measuring device and the camera”);
a second time correction parameter (Liu- ¶0021, at least discloses Optimize the initial value of the first internal parameter of the distance measuring device and the initial value of the second internal parameter of the camera to obtain the optimized first target internal parameter of the distance measuring device and the optimized second target of the camera Internal reference);
a second screening parameter, wherein the second screening parameter is a second projection error threshold (Liu- ¶0021, at least discloses Optimize the initial value of the first internal parameter of the distance measuring device and the initial value of the second internal parameter of the camera to obtain the optimized first target internal parameter of the distance measuring device and the optimized second target of the camera Internal reference).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Frueh/Zhang to incorporate the teachings of Liu, and apply the reprojection error function model into Frueh/Zhang’s teachings for performing screening of sampling information for the plurality of pairs of sampling information based on a first screening parameter, wherein the first screening parameter is a first projection error threshold; aligning the plurality of pairs of sampling information based on a second time correction parameter; and performing screening of sampling information for the plurality of pairs of sampling information based on a second screening parameter, wherein the second screening parameter is a second projection error threshold, and the first projection error threshold is greater than the second projection error threshold.
Doing so would provide the accuracy of the 3D measurement results, thereby affecting the calibration accuracy of the external parameters between multiple sensors.
Regarding claim 8, Frueh in view of Zhang and Liu, discloses the method of claim 7, and further discloses wherein a manner of screening the sampling information is a random sample consensus algorithm (Zhang- ¶0108-0110, at least disclose Preferably, the point coordinates of the first circle around the first VR handle in the two-dimensional image may be curve-fitted using a RANdom Sample Consensus (RANSAC) algorithm to obtain the coordinates of the A circle corresponding to the elliptic expression. Among them, the RANSAC algorithm estimates the parameters of an elliptical expression from a set of observation data sets containing "outliers" […] The input to the RANSAC algorithm is a set of observations and an initial elliptic expression. The parameters in the elliptic expression are the initial parameters. RANSAC fits elliptic expressions by repeatedly selecting a random set of points in the point coordinates on the first ring identified by RCNN ).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Frueh/Liu to incorporate the teachings of Zhang, and apply the RANSAC algorithm into Frueh/Liu’s teachings in order a manner of screening the sampling information is a random sample consensus algorithm.
Doing so would be able to obtain sampling information in multiple dimension directions and improve the calibration accuracy.
The system of claims 20-21 are similar in scope to the functions performed by the method of claims 7-8 and therefore claims 20-21 are rejected under the same rationale.
9. Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Frueh in view of Zhang, further in view of Liu, still further in view of Mitsunami et al. (machine translation of JP-2017059049-A with citation below, hereinafter “Mitsunami”)
Regarding claim 9, Frueh in view of Zhang and Liu, discloses the method of claim 7, and further discloses wherein the aligning the plurality of pairs of sampling information based on the second time correction parameter (see Claim 7 rejection for detailed analysis) comprises:
determining a plurality of candidate second time correction parameters in a second time correction parameter range (Liu- ¶0021, at least discloses Optimize the initial value of the first internal parameter of the distance measuring device and the initial value of the second internal parameter of the camera to obtain the optimized first target internal parameter of the distance measuring device and the optimized second target of the camera Internal reference), and adjusting a sampling time parameter of the two-dimensional detection data and a sampling time parameter of the three-dimensional detection data comprised in the plurality of pairs of sampling information respectively based on each of the candidate second time correction parameters (Frueh- Fig. 17 and ¶0117-0118, at least disclose “The calibration is performed on the basis of distinguishing features of the HMD in the different orientations […] For example, the calibration can be performed on the basis of an Aruco marker 1745 that is affixed to a surface of the HMD […] the calibration can be performed on the basis of a logo or other specific geometric features of the HMD such as edges, corners, cable connectors, and the like […] The camera 1705 captures images […] of the HMD in the corresponding orientations […] For each feature detected in the captured images […] the processor 1708 is configured to determine a 3-D location of the feature in the HMD coordinate system using a pose of the HMD for the corresponding orientations […] and the measured offsets. As discussed herein, the processor 1708 is able to acquire the pose information for the HMD in a telemetry stream received from the HMD. The processor 1708 is also able to identify pixel locations of the features in the captured images […] The 3-D location and pixel locations of the feature form a 2-D/3-D point pair. The processor 1708 is therefore able to generate a set of 2-D/3-D point pairs from the captured images […] and the corresponding pose information. The set of 2-D/3-D point pairs is then used to determine a projection matrix that projects the 3-D features onto the 2-D image plane of the camera 1705; ¶0121, at least discloses “At block 1810, the processor determines 3-D locations of the HMD features in the captured image in HMD coordinates. The 3-D locations of the HMD features can be determined by computing a central point (or other reference point) of the HMD using pose information in a telemetry stream provided to the processor by the HMD” Regarding what this "position information in a telemetry stream provided by the HMD" and whether it comprises the claimed "acquiring ... a pose of a target device": [0027] describes that the HMD can be such as an an Oculus Rift or HTC Vive provides 6Degree of Freedom (6DoF) pose data posture. And [0131] notes that, "The sensors 940, 945 in the electronic device 2005 are able to generate information indicating the six degree-of-freedom (6DoF) pose of the electronic device 2005, which includes a three-dimensional position of the electronic device 2005 and a three-dimensional orientation of the electronic device 2005” [Wingdings font/0xE0] Obtaining the sampling time parameter of the two-dimensional detection data as the image acquisition moment when the image acquisition device acquires the two dimensional detection data, and the sampling time parameter of the three-dimensional detection data as the detection moment when the virtual reality device detects the three-dimensional detection data according to the actual needs;
updating the sampling information based on at least two of the plurality of three-dimensional detection data corresponding to each of the candidate second time correction parameters (Frueh- ¶0118, at least discloses “the set of 2-D/3-D point pairs includes images taken from different positions by a moving camera 1705 and the locations of the camera when the images were acquired”); and
screening the second time correction parameter from the plurality of candidate
second time correction parameters, wherein the second time correction parameter is a candidate second time correction parameter with a largest number of sampling information corresponding to the plurality of candidate second time correction parameters (Liu- ¶0021, at least discloses “Optimize the initial value of the first internal parameter of the distance measuring device and the initial value of the second internal parameter of the camera to obtain the optimized first target internal parameter of the distance measuring device and the optimized second target of the camera Internal reference”).
The prior art does not explicitly disclose, but Mitsunami discloses
a largest number of inlier sampling information (Mitsunami- ¶0005, at least discloses “In robust estimation, feature point pairs whose projection error due to the transformation matrix is below a threshold are called "inliers," and all others are called "outliers." In RANSAC, the higher the proportion of outliers relative to the number of input feature points, the more iterations are required. Furthermore, the more elements in the transformation matrix to be estimated, the more iterations are required. Therefore, Patent Document 1 proposes a technique for speeding up processing by RANSAC by improving the matching of input feature points”; ¶0032-0033, at least disclose ” Here, "inlier" refers to a pair of feature points where the distance (projection error) between the coordinates obtained by transforming the feature point coordinates (start coordinates) of the preceding image frame using a transformation matrix (end coordinates) and the feature point coordinates of the subsequent image frame is below a threshold […] In step S204, the transformation matrix estimation unit 107 compares the number of inliers determined in step S203 with the maximum number of inliers in one or more transformation matrices that have already been determined. Then, if the number of inliers for the transformation matrix currently under consideration reaches a new record high, the process proceeds to step S205. On the other hand, if the number of inliers for the transformation matrix currently under consideration does not exceed the highest value, the process proceeds to step S206 […] the transformation matrix estimation unit 107 updates (sets) the transformation matrix in which the number of inliers has reached its highest value as the best transformation matrix that best represents the positional shift between image frames”).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Frueh/Zhang/Liu to incorporate the teachings of Mitsunami, and apply the number of inliers into Frueh/Zhang/Liu’s teachings for screening the second time correction parameter from the plurality of candidate second time correction parameters, wherein the second time correction parameter is a candidate second time correction parameter with a largest number of inlier sampling information corresponding to the plurality of candidate second time correction parameters.
Doing so would provide a technique for correcting the shaking of temporally consecutive image frames is alignment by matching feature points in the image.
Conclusion
10. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. They are as recited in the attached PTO-892 form.
11. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL LE whose telephone number is (571)272-5330. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL LE/Primary Examiner, Art Unit 2614
Prosecution Insights