Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4 and 16-17 are rejected under 35 U.S.C. sec. 102(a)(2) as being anticipated by International Patent Pub. No.: WO 2007/030026 A1 to Volkenburg et al. that was filed in 2006.
PNG
media_image1.png
790
776
media_image1.png
Greyscale
In regard to claim 1, and 16, Valkenburg discloses ‘...1. A system for providing 3D surveying of an environment, the system comprising a mobile reality capture device configured to be carried and moved during generation of 3D measurement data, wherein the system comprises: (see range sensor 11 and position and orientation sensor 13 and the CPU 14 and texture sensor that can take images of the object)
a 3D surveying unit arranged on the mobile reality capture device and configured to provide the generation of the 3D measurement data for carrying out a spatial 3D measurement of the environment relative to the mobile reality capture device, wherein the 3D surveying unit is configured to provide the spatial 3D measurement with a field-of view
of 360 degrees around a first device axis (See page 3-5 and systems reconstruct a 3D scene or object based on analysis of multiple overlapping 2D images. Provided common features are visible and identified in the images and camera calibration parameters are known or determined, it is possible to extract 3D metric scene or object information. In some cases, the cameras are pre- calibrated. In other cases, self-calibration is attempted based on the image matches. Fixed station scanners scan a scene from a fixed location. Typically, fixed station scanners are arranged to scan a modulated laser beam in two dimensions and acquire range information by measuring the phase-shift of the reflected modulated laser light or the time-of-flight of a reflected laser pulse. By panning the scanner through 360°, it is possible to produce a 360° panoramic range map of the scene. To scan a complete scene often requires moving the fixed station scanner to a number of different scanning locations. Depending on the size of scene, scanning time is typically 10-30 minutes. Some fixed station scanners also comprise a digital camera that is arranged to capture color information for each surface point in the scan of the scene so that dual color and range images can be generated. Other fixed station scanners incorporate multiple lasers to allow acquisition of color as well as range information.)
PNG
media_image2.png
726
648
media_image2.png
Greyscale
and 120 degrees around a second device axis perpendicular to the first device axis, (see Fig. 2 where the sensor can capture 360 degrees in the x-y plane and 360 degrees in the x-z and y-z plane and has sensors on all sides)
an inertial measurement unit comprising sensors including accelerometers and/or (optionally)
gyroscopes, and being configured to continuously generate IMU data related to a pose (see FIG. 3 where the device has a compass and gyroscopic device; In its most generic form, the scanner 10 can have a number of different position and orientation subsystems, which collectively make up the position and orientation sensor 13. As mentioned, the subsystems may be optical, electromagnetic, GPS, inertial or the like. The or each position subsystem provides an estimate of the position of its mobile position sensor, mounted to or within the mobile scanner 10, with respect to its assigned reference coordinate system, defined as the position of CSMp in CSWp. For example, a GPS sensor would provide its position with respect to the earth's latitude and longitude coordinate system plus altitude. Similarly, the or each orientation subsystem provides an estimate of the orientation of its mobile orientation sensor, mounted to or within the mobile scanner 10, with respect to its assigned reference coordinate system, defined as the orientation of CSMo in CSWo. For example, a gyro-based orientation sensor provides its orientation with respect to the defined inertial frame of reference. A magnetic compass defines its orientation with respect to earth's magnetic field. It will be appreciated that some sensors may estimate both position and orientation. For example, an electromagnetic motion tracking system may be utilised. Such a system often utilises receivers, located on or within the mobile scanner 10, that provide their position and orientation information relative to a fixed-point reference source transmitting station.)
and/or acceleration of the mobile reality capture device, and (See FIG. 5 where the device has an accelerometer; The preferred form portable 3D scanning system, comprising the hand-held mobile scanner 10, reference targets 20, and laptop 15, may be transported easily to the crash scene by the operator 19 in a carry-case or the like. To scan the scene, the operator 19 starts by placing a number of reference targets 20 in suitable locations around the scene. The reference targets 20 provide a framework for the optical tracking device of the position and orientation sensor 13 onboard the scanner 10 and in the preferred form system the location of the reference targets 20 define a local reference frame within which the scanner 10 can operate. The location of the reference targets 20 can be determined arbitrarily and randomly by the operator 19. The scanner 10 does not need to be pre-programmed with the reference target 20 locations as the target locations are obtained automatically by the scanner 10 in operation. The reference targets 20 can be tripod mounted or otherwise mounted in convenient locations, for example they may be placed on objects in the scene. The number of reference targets 20 is flexible and is dictated by the size and complexity of the scene. The key requirement is optimum visibility from the scanner 10 in operation. Not all the reference targets 20 need to be visible at all times, but to maintain position and orientation sensing integrity at least three must be visible at any one point in time. As previously mentioned, the position and orientation sensor 13 may comprise a number of position and orientation subsystems to enhance accuracy and reliability. Therefore, the optical tracking device may be augmented with other inertial sensors, such as gyros, accelerometers, inclinometers or the like, to maintain spatial position and orientation sensing capability in the event of target visibility dropout. )
a simultaneous localization and mapping unit configured to carry out a simultaneous
localization and mapping process comprising generation of a map of the environment and
determination of a trajectory of the mobile reality capture device in the map of the environment,
wherein the system comprises a motion state tracker configured to use motion data
regarding a movement of the mobile reality capture device to determine a motion pattern
of the mobile reality capture device, (see FIG. 6 where the device can include a solution to provide a motion pattern of the scene and the pose of the device; Once the above process has been carried out, the problem can be stated as: given a set of labelled lines in CSM captured from N positions, recover the position of the K targets in CSW. The solution involves three steps:
(1) Calculate initial estimates of the camera group pose for n = 1, . . . N, using a closed form algorithm; (2) Calculate accurate estimates of the camera group pose by non-linear minimization; and
(3) Reconstruct the target positions in CSW by triangulation using the set of labelled lines and accurate camera group pose estimates.
Referring to step (1), this is carried out to avoid ambiguities in the non-linear minimization that occurs in step (2). In particular, the initial estimates of the camera group pose at each of the N positions assist in avoiding local minima of the objective function used in performing the non-linear minimization to calculate accurate estimates of the camera group pose. There are various closed form algorithms that may be utilised to calculate the initial estimates and each algorithm makes certain assumptions. By way of example, one closed form algorithm calculates the initial estimates based on approximating the camera group by a more idealised omnidirectional camera in which all the optical centres are coincident and positioned at the origin of the CSM. This allows many standard results to be applied directly and the results may be expressed in terms of geometric algebra. The algorithm may then employ the well known essential transformation to determine the relative pose between each of the N positions. The relative pose may be expressed in terms of direction and orientation, but not distance as a unit translation is assumed. The essential transformation considers the labelled lines at each of the N positions of the CSM and obtains estimates of the relative pose (with unit translation) between the positions of the CSM. These estimates can then be transformed into CSW to give initial estimates of the relative direction and orientation between the positions of the CSM and known distances (yardsticks) can be used to rescale the estimates. For example, yardsticks might be obtained via measuring the actual distance between two reference targets.
Referring to step (2) accurate estimates of the camera group CSM pose at each of the N positions are calculated using a non-linear minimisation algorithm and the initial estimates calculated in step (1). By way of example, an objective (error) function and its gradient is utilised so that the problem can be formulated as a standard non-linear optimisation. The non-linear optimisation does not assume the idealised omnidirectional camera approximation. The lines associated with a given target K will nearly intersect. Image noise and quantisation, calibration errors, camera modelling errors etc will prevent them from intersection exactly. The objective (error) function is utilised to minimise the dispersion of the lines about their nominal intersection point. The objective function, constraints such as yardsticks, the gradients of the objective function, and the initial estimates enable the problem to be formulated as a standard optimisation with barrier functions, or a constrained optimisation. The results of the optimisation provide accurate estimates of the camera group CSM pose at each of the N positions in CSW.
Referring to step (3), the positions of the K targets in CSW may then be triangulated using the set of labelled lines and accurate camera group pose estimates at each of the N positions. This involves mapping the labelled lines into CSW and then triangulating the lines associated with each of the K targets. During steps (l)-(3), a temporary assumption is made that the origin of CSW is identified by one of the N positions of the camera group CSM. This can now be dropped and the true CSW can be constructed out of the targets and the target positions mapped into this CSW.
As mentioned, once the auto-calibration method has obtained the target locations, the scanner can be arranged to diagnose geometrically ill-conditioned reference target configurations and provide feedback to the user on whether the optical tracking device can provide accurate pose information for a particular distribution and spread of targets within a scene. The target location feedback may advise the user to, for example, add a target or rearrange the targets for a better spread across the scene. For example, the target location feedback algorithm may involve calculating some measure of dispersion of the unit direction vectors from the direction sensors to each target, where a high dispersion represents a good target configuration and a low dispersion indicates a poor target organization.)
wherein, if (optionally) a determined motion pattern corresponds
to a defined movement category associated to an environment-specific measurement
movement of the mobile reality capture device, the system is configured to automatically
perform a derivation of an expected motion pattern of the mobile reality capture device for
the environment-specific measurement movement, and to carry out a comparison of the determined motion pattern and the expected motion pattern, (see page 31-32 where the determined motion pattern fails to be expected and the auto correction algorithm can rearrange the targets; ; As mentioned, once the auto-calibration method has obtained the target locations, the scanner can be arranged to diagnose geometrically ill-conditioned reference target configurations and provide feedback to the user on whether the optical tracking device can provide accurate pose information for a particular distribution and spread of targets within a scene. The target location feedback may advise the user to, for example, add a target or rearrange the targets for a better spread across the scene. For example, the target location feedback algorithm may involve calculating some measure of dispersion of the unit direction vectors from the direction sensors to each target, where a high dispersion represents a good target configuration and a low dispersion indicates a poor target organization.
In the pose tracking method, the camera group CSM is moved continuously in CSW and a continuous sequence of target image sets is acquired, each containing M images. For each image set (frame event), the set of observed lines in CSM, corresponding to the set of visible targets, can be determined. The problem can now be stated as: given the set of lines in CSM for each frame capture event, estimate the pose of the camera group. The camera group pose estimation is accomplished by repeating the following steps:
(1) Predict the current pose based on the previous pose estimates using extrapolation;
(2) Associate target labels with each line in CSM; and
(3) Update the current pose prediction using the targets and lines using a non-linear algorithm.
Before steps (l)-(3) are traversed, the pose tracking method implements an initial bootstrapping process to provide an initial pose estimate. The initial pose estimate is obtained by maintaining the camera group CSM at an initial position and then acquiring a sequence of images of the targets as they are selectively activated according to a pattern. By way of example, this process may be similar to that carried out by the auto- calibration process at each of the N positions. Analysis of the sequence of images (frame events) yields a set of lines in CSM, corresponding to the set of visible targets, labelled with their target identifiers. As the target positions are now known, an initial pose estimate of the camera group can be determined using triangulation.)
wherein the system is configured to provide feedback regarding the comparison of the determined motion pattern and the expected motion pattern, wherein the mobile reality
capture device is configured to take into account the feedback to automatically perform an action associated with the expected motion pattern” (see page 31-34 where the motion pattern does not follow what is expected and a revision is provided; After the boot-strapping process, steps (l)-(3) are repeated rapidly for each frame event to provide continuous estimates of the camera group in CSW. As mentioned, during steps (l)-(3) all reference targets are activated during each frame event and hence the observed lines to each of the targets in CSM are not labelled.
Referring to step (1), extrapolation is used to predict the current pose based on previous pose estimates. Referring to step (2), the predicted pose and previous set of labelled lines are used to get a prediction of the next position of the labelled lines. The observed set of lines, whose identifiers are unknown, are associated with target labels or identifiers based on the predicted labelled lines determined. This is essentially done by matching the predicted labelled lines with the closest observed unknown lines. Referring to step (3), the labelled set of observed lines is now utilised to provide a camera group pose estimate and update the current pose prediction using a non-linear algorithm for the next iteration of steps (l)-(3). Reverting back to the boot-strapping process, this provides an initial prediction of the current pose for step (1) of the first iteration of steps (l)-(3). The boot-strapping process is not utilised after the first iteration.
It will be appreciated that all reference targets do not necessarily have to be activated for each frame. The pose tracking method could utilise a patterned sequence of activation in alternative forms of the tracking method.
As the optical tracking device is onboard the scanner 10, the camera group pose reflects the position and orientation of the scanner 10.
The preferred form reference targets 20 comprise a powered light source that emits visible electromagnetic radiation for detection by the cameras 22 of the optical tracking device. For example, the light sources may be LEDs, or more particularly high-power blue LEDs. The LEDs may be operated to glow continuously or may be pulsed at a particular switching frequency. In the preferred form, the LEDs of the reference targets 20 are pulsed in synchronism with the direction sensor camera shutter. This allows a much higher LED intensity than that possible with continuous operation and improves LED contrast with ambient since each camera is only acquiring light when the LEDs are operating. Further, the direction sensor cameras may utilise optical filtering to further enhance the contrast with ambient. The preferred form LEDs are individually addressed to provide facility for unique identification by direction sensors. )
Valkenburg discloses ‘...2. The system according to claim 1, wherein the system comprises a database comprising
a plurality of defined motion patterns, each defined motion pattern being associated with an environment-specific measurement movement of the mobile reality capture device,
particularly wherein each defined motion pattern is either pre-defined or (optionally) user-defined,
wherein the database is used for a categorization of the determined motion pattern and/or (optionally)
the derivation of the expected motion pattern”. (see page 30-1 and 31-34 where the motion pattern does not follow what is expected and a revision is provided; After the boot-strapping process, steps (l)-(3) are repeated rapidly for each frame event to provide continuous estimates of the camera group in CSW. As mentioned, during steps (l)-(3) all reference targets are activated during each frame event and hence the observed lines to each of the targets in CSM are not labelled.
Referring to step (1), extrapolation is used to predict the current pose based on previous pose estimates. Referring to step (2), the predicted pose and previous set of labelled lines are used to get a prediction of the next position of the labelled lines. The observed set of lines, whose identifiers are unknown, are associated with target labels or identifiers based on the predicted labelled lines determined. This is essentially done by matching the predicted labelled lines with the closest observed unknown lines. Referring to step (3), the labelled set of observed lines is now utilised to provide a camera group pose estimate and update the current pose prediction using a non-linear algorithm for the next iteration of steps (l)-(3). Reverting back to the boot-strapping process, this provides an initial prediction of the current pose for step (1) of the first iteration of steps (l)-(3). The boot-strapping process is not utilised after the first iteration.
It will be appreciated that all reference targets do not necessarily have to be activated for each frame. The pose tracking method could utilise a patterned sequence of activation in alternative forms of the tracking method.
As the optical tracking device is onboard the scanner 10, the camera group pose reflects the position and orientation of the scanner 10.
The preferred form reference targets 20 comprise a powered light source that emits visible electromagnetic radiation for detection by the cameras 22 of the optical tracking device. For example, the light sources may be LEDs, or more particularly high-power blue LEDs. The LEDs may be operated to glow continuously or may be pulsed at a particular switching frequency. In the preferred form, the LEDs of the reference targets 20 are pulsed in synchronism with the direction sensor camera shutter. This allows a much higher LED intensity than that possible with continuous operation and improves LED contrast with ambient since each camera is only acquiring light when the LEDs are operating. Further, the direction sensor cameras may utilise optical filtering to further enhance the contrast with ambient. The preferred form LEDs are individually addressed to provide facility for unique identification by direction sensors. )
Valkenburg discloses “... 3. The system according to claim 2, wherein the system is configured to establish a data connection with a remote server computer and to provide motion data to the remote server
computer, wherein the system is configured (see FIG. 1 where the scanner can be local processor computing the algorithm or via a remote computing device and the scanner 10 communicates wirelessly with an external device 15, such as a portable laptop, PDA or other mobile computer. In particular, the control system 14 onboard the scanner 10 is arranged to transfer scanned data to the external computer 15 for post-processing and storage. In one form, the external computer 15 may perform some control system functions for the scanner 10, such as communicating with reference targets 20 or controlling scanner 10 settings remotely via a wireless link. As mentioned, the control system for the scanner 10 can be partially external to the scanner itself in some embodiments. Furthermore, it will be appreciated that communication between the scanner 10, external devices 15, and reference targets 20 may be via wire if desired.)
to detect typical behavior of a carrier of the mobile reality capture device from the motion data and to send corresponding motion data to the remote server computer, wherein the sent data is dedicated to update pre-defined motion patterns stored at the remote server
computer, and to receive updated pre-defined motion patterns from the remote server computer”. (see page 30-1 and 31-34 where the motion pattern does not follow what is expected and a revision is provided; After the boot-strapping process, steps (l)-(3) are repeated rapidly for each frame event to provide continuous estimates of the camera group in CSW. As mentioned, during steps (l)-(3) all reference targets are activated during each frame event and hence the observed lines to each of the targets in CSM are not labelled.
Referring to step (1), extrapolation is used to predict the current pose based on previous pose estimates. Referring to step (2), the predicted pose and previous set of labelled lines are used to get a prediction of the next position of the labelled lines. The observed set of lines, whose identifiers are unknown, are associated with target labels or identifiers based on the predicted labelled lines determined. This is essentially done by matching the predicted labelled lines with the closest observed unknown lines. Referring to step (3), the labelled set of observed lines is now utilised to provide a camera group pose estimate and update the current pose prediction using a non-linear algorithm for the next iteration of steps (l)-(3). Reverting back to the boot-strapping process, this provides an initial prediction of the current pose for step (1) of the first iteration of steps (l)-(3). The boot-strapping process is not utilised after the first iteration.
It will be appreciated that all reference targets do not necessarily have to be activated for each frame. The pose tracking method could utilise a patterned sequence of activation in alternative forms of the tracking method.
As the optical tracking device is onboard the scanner 10, the camera group pose reflects the position and orientation of the scanner 10.
The preferred form reference targets 20 comprise a powered light source that emits visible electromagnetic radiation for detection by the cameras 22 of the optical tracking device. For example, the light sources may be LEDs, or more particularly high-power blue LEDs. The LEDs may be operated to glow continuously or may be pulsed at a particular switching frequency. In the preferred form, the LEDs of the reference targets 20 are pulsed in synchronism with the direction sensor camera shutter. This allows a much higher LED intensity than that possible with continuous operation and improves LED contrast with ambient since each camera is only acquiring light when the LEDs are operating. Further, the direction sensor cameras may utilise optical filtering to further enhance the contrast with ambient. The preferred form LEDs are individually addressed to provide facility for unique identification by direction sensors. )
Valkenburg discloses “... 4. The system according to claim 1, wherein the expected motion pattern provides
a nominal orientation or a sequence of nominal orientations of the mobile reality capture device with regard to three mutually perpendicular axes of rotation of the mobile
reality capture device, and/or (optionally)
a nominal relative position change or a sequence of nominal relative position changes of the mobile reality capture device with respect to a current position of the mobile reality capture device with regard to three mutually perpendicular spatial axes”. (See FIG. 6 where the sensors can be placed on the top, bottom and sides and capture 360 degrees; see page 31-34 and FIG. 6 where To describe the algorithms, we assume that the optical tracking device utilises a set of M calibrated cameras that have a fixed spatial relationship relative to each other, for example they are rigidly coupled together. The set of cameras is referred to as the camera group. The cameras are organised to approximate an omnidirectional camera with their optical centres as coincident as physically possible, and the image planes providing maximum coverage with as little overlap as possible. The geometric relationship between each camera is known by means of a group calibration process, so the camera group can be characterised by a single moving coordinate system denoted CSM. For example, CSM may be identified with the camera coordinate system of one of the cameras. The cameras are synchronised so that a single frame capture event will grab M images. A world coordinate system, denoted CSW, is defined in terms of K stationary reference targets 20 located in a scene 23 (e.g. one target is at the origin, another on the x-axis and another in the xy-plane).
In both the auto-calibration and pose tracking methods, a set of M images of the reference targets 20 is captured (one from each camera) at each camera group CSM position. Not all the reference targets 20 will necessarily be visible as the sensors do not give complete coverage and the targets may also be occluded. Some reference targets 20 may also be visible in more than one image because of small amounts of sensor overlap. A target pixel location is extracted using a sub-pixel estimator then back-projected through the camera model to get a ray in the camera coordinate system. The ray is then rotated and translated to provide a line in CSM.
In the auto-calibration method, the camera group CSM is moved to N positions 24 in CSW and for each position a sequence of images of targets is acquired with targets selectively activated according to a pattern. Analysis of the sequence of images (frame events) yields a set of lines in CSM, corresponding to the set of visible targets, labelled with their target identifiers.
By way of example only, the camera group may have six outward looking cameras (M=6) as previously described and there may be 32 reference targets in the scene. At each of the N positions in the scene the camera group may be arranged to rapidly capture 33 frame events (sets of 6 images) as the reference targets are selectively activated to sequence through a pattern. The pattern may, for example, involve selectively activating one target at a time for the first 32 frame events and then activating all 32 targets for the 33rd frame event. As previously mentioned, analysis of the 33 frame events yields a set of lines in the CSM, corresponding to the set of visible targets, labelled with their target identifiers.
It will be appreciated that various pattern sequences may alternatively be utilized. For example, binary coded sequences or other coded sequences could be implemented to selectively activate the targets as the camera group captures frame events at each of the N positions. Further, the number of frame events required at each of the positions is inherently related to the type of sequenced pattern generated by the reference targets. More efficient pattern sequences will require fewer frame events at each of the N positions to yield the required set of labelled lines. It will also be appreciated that the number of reference targets may be varied according to the nature of the scene. Ultimately, the auto-calibration method may work on considerably fewer targets. Further the number and location of the N positions may vary and can be selected at random.
Once the above process has been carried out, the problem can be stated as: given a set of labelled lines in CSM captured from N positions, recover the position of the K targets in CSW. The solution involves three steps:
(1) Calculate initial estimates of the camera group pose for n = 1, . . . N, using a closed form algorithm; (2) Calculate accurate estimates of the camera group pose by non-linear minimization; and
(3) Reconstruct the target positions in CSW by triangulation using the set of labelled lines and accurate camera group pose estimates.
Referring to step (1), this is carried out to avoid ambiguities in the non-linear minimization that occurs in step (2). In particular, the initial estimates of the camera group pose at each of the N positions assist in avoiding local minima of the objective function used in performing the non-linear minimization to calculate accurate estimates of the camera group pose. There are various closed form algorithms that may be utilised to calculate the initial estimates and each algorithm makes certain assumptions. By way of example, one closed form algorithm calculates the initial estimates based on approximating the camera group by a more idealised omnidirectional camera in which all the optical centres are coincident and positioned at the origin of the CSM. This allows many standard results to be applied directly and the results may be expressed in terms of geometric algebra. The algorithm may then employ the well known essential transformation to determine the relative pose between each of the N positions. The relative pose may be expressed in terms of direction and orientation, but not distance as a unit translation is assumed. The essential transformation considers the labelled lines at each of the N positions of the CSM and obtains estimates of the relative pose (with unit translation) between the positions of the CSM. These estimates can then be transformed into CSW to give initial estimates of the relative direction and orientation between the positions of the CSM and known distances (yardsticks) can be used to rescale the estimates.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 5 is rejected under 35 U.S.C. sec. 103 as being unpatentable as obvious in view of International Patent Pub. No.: WO 2007/030026 A1 to Volkenburg et al. that was filed in 2006 and in view of U.S. Patent No.: US10907940B1 to Parker.
Valkenburg is silent but Parker teaches “...5. The system according to claim 1, wherein a correspondence of the determined motion
pattern to the defined movement category and/or (optionally) the derivation of the expected motion pattern is/are provided by a machine learning algorithm, which comprises processing of the motion data by a Kalman filter for an estimation of an attitude parameter, and particularly a velocity parameter, of the mobile reality capture device, wherein the processing of the motion data is carried out in sections for time the motion data is carried out in a rolling fashion by continuously processing a continuously generated time series of the motion data.” (see col. 23, line 10 to col.. 24, line 15 where the heading, speed and classification and bearing of the target can be fed into the machine learning to identify the movement category and the expected motion pattern of the target and the heading and bearing and the speed of the target to classify the target using the sensors and the machine learning algorithm)”.
It would have been obvious for one of ordinary skill in the art to combine the teachings of PARKER with the disclosure of VOLKENBURG with a reasonable expectation of success since PARKER teaches that a time series of motion data including heading and bearing data and a speed of the object can be provided as inputs to the machine learning model. The model can then classify and recognize the target for further action and in this case to continue monitoring the drone or object of interest in a more focused manner or to intervene. See abstract and Col. 23, line 1 to col. 24, line 44 of PARKER.
Claim 6 is rejected under 35 U.S.C. sec. 103 as being unpatentable as obvious in view of International Patent Pub. No.: WO 2007/030026 A1 to Volkenburg et al. that was filed in 2006 and in view of U.S. Patent No.: US10907940B1 to Parker and in view of United States Patent Application Pub. No.: US20150316383A1 to Donikian filed in 2015.
Valkenburg is silent but Parker teaches “...6. The system according to claim 5, wherein the correspondence of the determined motion pattern to the defined movement category and/or the derivation of the expected motion pattern is/are provided by taking into account a feature extraction step, which provides detection of a signal feature out of a plurality of different signal features, wherein each of the signal features is indicative of a defined environment-specific (See Col. 6, lines 10-47 where the device can provide video and sensor data and LIDAR data to provide a feature extraction and classification of the threat to eliminate the threat via a machine learning processor)
It would have been obvious for one of ordinary skill in the art to combine the teachings of PARKER with the disclosure of VOLKENBURG with a reasonable expectation of success since PARKER teaches that a time series of motion data including heading and bearing data and a speed of the object can be provided as inputs to the machine learning model. The model can then classify and recognize the target for further action and in this case to continue monitoring the drone or object of interest in a more focused manner or to intervene. See abstract and Col. 23, line 1 to col. 24, line 44 of PARKER.
The primary reference is silent but DONIKIAN teaches “...measurement movement out of a plurality of defined environment-specific measurement movements, and is used for the estimation of the attitude parameter, and particularly the velocity parameter,
wherein the feature extraction step 1s provided by a deep learning algorithm configured to learn the signal features independently or wherein the feature extraction step is provided by computing the motion data with defined statistics in the frequency or time domain of the motion data”. (see claims 1-7 where the attitude of the object can be determined from a machine learning model and from different models) (see paragraph 9, and 28-30 where the neural network model can provide the attitude and the velocity of the object from an INS sensor and barometric and other sensors)”.
It would have been obvious for one of ordinary skill in the art to combine the teachings of DONIKIAN with the disclosure of VOLKENBURG with a reasonable expectation of success since DONIKIAN teaches that a neural network device can receive a number of sensor inputs. Based on these sensor inputs that can include velocity of the object, INS data of the object, pressure and attitude data the neural network can classify the data and identity the object based on the motion data and the neural network classification and the motion data can be provided to a model for improved further control. See abstract and claims 1-10.
Claims 7 and 9 are rejected under 35 U.S.C. sec. 103 as being unpatentable as obvious in view of International Patent Pub. No.: WO 2007/030026 A1 to Volkenburg et al. that was filed in 2006 and in view of \United States Patent Application Pub. No.: US20150316383A1to Donikian filed in 2015.
PNG
media_image3.png
740
584
media_image3.png
Greyscale
The primary reference is silent but DONIKIAN teaches “..7. The system according to claim 1, wherein the system is configured to analyze motion data in order to generate a movement model that takes into account parameters of a range of motion for a relative movement of the mobile reality capture device when it is carried and aligned by the carrier and/or (optionally) a weight-distribution of a combination of the mobile reality capture device with a companion device and/or (OPTIONALLY) the carrier, wherein the movement model is taken into account for at least one of a providing of a correspondence of the determined motion pattern to the defined movement category,(See paragraph 84) (optionally) the derivation of the expected motion pattern, and the comparison of the determined motion pattern with the expected motion pattern, wherein a center of mass and a moment of inertia of the combination of the mobile reality capture device with the companion device and/or the carrier is/are determined”. (see paragraph 28-34 where the motion model can provide 1. Velocity of the object and 2. Attitude of the object over time and based on the INS device and sensors and using a neural network computing device)
It would have been obvious for one of ordinary skill in the art to combine the teachings of DONIKIAN with the disclosure of VOLKENBURG with a reasonable expectation of success since DONIKIAN teaches that a neural network device can receive a number of sensor inputs. Based on these sensor inputs that can include velocity of the object, INS data of the object, pressure and attitude data the neural network can classify the data and identity the object based on the motion data and the neural network classification and the motion data can be provided to a model for improved further control. See abstract and claims 1-10.
Claim 8 is rejected under 35 U.S.C. sec. 103 as being unpatentable as obvious in view of International Patent Pub. No.: WO 2007/030026 A1 to Volkenburg et al. that was filed in 2006 and in view of \United States Patent Application Pub. No.: US20150316383A1to Donikian filed in 2015 and in view of Chinese Patent Application Pub. NO.: to Shandong.
Shandong teaches “...8. The system according to claim 7, wherein the mobile reality capture device comprises a calibration functionality based on a set of pre-defined control movements of the mobile reality capture device to be carried out by the carrier, wherein motion data measured during the control movements are analyzed in order to generate the movement model, wherein the parameters of the range of motion provide information on a length of a boom-component carrying the mobile reality capture device”. (see abstract and claims 1-4 where a control movement of the boom can be determined and a model is creating to adjust the boom in the range of motion based on the length)
It would have been obvious for one of ordinary skill in the art to combine the teachings of SHANDONG with the disclosure of VOLKENBURG with a reasonable expectation of success since SHANDONG teaches that a boom can be tracked a motion of the boom can be predicted for a range of motion of the boom based on the length of the boom and the size. This can be used for estimating the motion of the boom in real time to prevent collisions and for improved control of the boom.
The primary reference is silent but DONIKIAN teaches “..9. The system according to claim 1, wherein:
the mobile reality capture device is configured to derive perception data, particularly from the 3D measurement data and/or from a sensor of the simultaneous localization and
mapping unit, wherein the perception data provide for a visual recognition of spatial features of the environment and for an evaluation of a spatial arrangement of the mobile
reality capture device relative to the spatial features,
the system is configured to analyze the perception data in order to provide a recognition of an environment-specific measurement situation with regard to a spatial
arrangement of the mobile reality capture device relative to spatial features in the environment, and to take into account the environment-specific measurement situation for deriving a
motion category of the determined motion pattern and/or for the derivation of the expected motion pattern. (see paragraph 84 and 28-34 where the motion model can provide 1. Velocity of the object and 2. Attitude of the object over time and based on the INS device and sensors and using a neural network computing device and where a motion model can be provided and input to the neural network for feature extraction)”.
It would have been obvious for one of ordinary skill in the art to combine the teachings of DONIKIAN with the disclosure of VOLKENBURG with a reasonable expectation of success since DONIKIAN teaches that a neural network device can receive a number of sensor inputs. Based on these sensor inputs that can include velocity of the object, INS data of the object, pressure and attitude data the neural network can classify the data and identity the object based on the motion data and the neural network classification and the motion data can be provided to a model for improved further control. See abstract and claims 1-10.
Claim 10 is rejected under 35 U.S.C. sec. 103 as being unpatentable as obvious in view of International Patent Pub. No.: WO 2007/030026 A1 to Volkenburg et al. that was filed in 2006 and in view of \United States Patent Application Pub. No.: US20150316383A1to Donikian filed in 2015 and in view of U.S. Patent No.: US5497430A to Sadovnik et al.
SADOVNIK et al. teaches “...10. The system according to claim 9, wherein: the system is configured to access a database comprising a set of geometric and/or (optionally)
semantic classes of spatial features with corresponding classification parameters for identifying the geometric and/or semantic classes by the perception data, of the geometric and/or semantic classes is associated with at least one of a rule regarding a minimum and/or a maximum distance between the mobile reality capture device and the corresponding spatial feature associated with that class, and a rule regarding a nominal relative orientation of the mobile reality
capture device to the corresponding spatial feature associated with that class, andthe system is configured to use the database comprising the set of geometric and/orsemantic classes to recognize the environment-specific measurement situation. (see abstract and claims 1-10 where the neural network can store a database of shapes for comparing the mobile device and the orientation to adjust the camera based on the neural network outputs; and see col. 6, line 10 to col. 7, line 55 where the device can determine that the object is a face or a jeep and then adjust the parameters of the camera accordingly)”.
It would have been obvious for one of ordinary skill in the art to combine the teachings of SADOVNIK with the disclosure of VOLKENBURG with a reasonable expectation of success since SADOVNIK teaches that an improved camera device and 3d scanner can be coupled to a neural network device. The neural network can classify the object as a vehicle or a human face based on a feature vector comparison. The object or vehicle can be rendered as a feature vector and then uploaded to the Bayesian artificial neural network. Then a classification that this is a face or a vehicle can be made and the camera can be adjusted and portion of the image can be panned, or brighten up to provide an automatic adjustment of the camera. See abstract and col. 6, line 1 to col. 7, line 49 and also claims 1-10.
Claims 11-12 are rejected under 35 U.S.C. sec. 103 as being unpatentable as obvious in view of International Patent Pub. No.: WO 2007/030026 A1 to Volkenburg et al. that was filed in 2006 and in view of \United States Patent Application Pub. No.: US20150316383A1to Donikian filed in 2015 and in view of U.S. Patent No.: US5497430A to Sadovnik et al.
SADOVNIK et al. teaches “...11. The system according to claim 9, wherein:
the system comprises an object detection algorithm based on machine learning, wherein the object detection algorithm is specifically configured to identify a spatial
constellation within the perception data,the spatial constellation is associated with a pre-defined sequence of motion states of the mobile reality capture device, particularly a sequence of relative orientations and/or distances between the mobile reality capture device and the spatial constellation, and the system is configured to take into account an identification of the spatial constellation by the object detection algorithm for the recognition of the environment specific
measurement situation, wherein the pre-defined sequence of motion states is taken into account for the derivation of the expected motion pattern”. (see abstract and claims 1-10 where the neural network can store a database of shapes for comparing the mobile device and the orientation to adjust the camera based on the neural network outputs; and see col. 6, line 10 to col. 7, line 55 where the device can determine that the object is a face or a jeep and then adjust the parameters of the camera accordingly and see col. 8, lines 1-55 where the camera can determine that the person is static based on the feature recognition and then adjust a brightness or that the moving vehicle is detected and then can pan the image as desired; Such an effective extractor would not require additional image processing for image centering, orientation correction, and scale adjustment.)”.
It would have been obvious for one of ordinary skill in the art to combine the teachings of SADOVNIK with the disclosure of VOLKENBURG with a reasonable expectation of success since SADOVNIK teaches that an improved camera device and 3d scanner can be coupled to a neural network device. The neural network can classify the object as a vehicle or a human face based on a feature vector comparison. The object or vehicle can be rendered as a feature vector and then uploaded to the Bayesian artificial neural network. Then a classification that this is a face or a vehicle can be made and the camera can be adjusted and portion of the image can be panned, or brighten up to provide an automatic adjustment of the camera. See abstract and col. 6, line 1 to col. 7, line 49 and also claims 1-10.
SADOVNIK et al. teaches “...12. The system according to claim 10, wherein: the system comprises an object detection algorithm based on machine learning, wherein the object detection algorithm is specifically configured to identify a spatial
constellation within the perception data, the spatial constellation is associated with a pre-defined sequence of motion states of the mobile reality capture device, particularly a sequence of relative orientations and/or distances between the mobile reality capture device and the spatial constellation, and the system is configured to take into account an identification of the spatial constellation by the object detection algorithm for the recognition of the environment specific measurement situation, wherein the pre-defined sequence of motion states is taken into account for the derivation of the expected motion pattern. (see abstract and claims 1-10 where the neural network can store a database of shapes for comparing the mobile device and the orientation to adjust the camera based on the neural network outputs; and see col. 6, line 10 to col. 7, line 55 where the device can determine that the object is a face or a jeep and then adjust the parameters of the camera accordingly and see col. 8, lines 1-55 where the camera can determine that the person is static based on the feature recognition and then adjust a brightness or that the moving vehicle is detected and then can pan the image as desired; Such an effective extractor would not require additional image processing for image centering, orientation correction, and scale adjustment.)
It would have been obvious for one of ordinary skill in the art to combine the teachings of SADOVNIK with the disclosure of VOLKENBURG with a reasonable expectation of success since SADOVNIK teaches that an improved camera device and 3d scanner can be coupled to a neural network device. The neural network can classify the object as a vehicle or a human face based on a feature vector comparison. The object or vehicle can be rendered as a feature vector and then uploaded to the Bayesian artificial neural network. Then a classification that this is a face or a vehicle can be made and the camera can be adjusted and portion of the image can be panned, or brighten up to provide an automatic adjustment of the camera. See abstract and col. 6, line 1 to col. 7, line 49 and also claims 1-10.
Claims 13-15 are rejected under 35 U.S.C. sec. 103 as being unpatentable as obvious in view of International Patent Pub. No.: WO 2007/030026 A1 to Volkenburg et al. that was filed in 2006 and in view of U.S. Patent No.: US5497430A to Sadovnik et al.
SADOVNIK et al. teaches “.13. The system according to claim 1, wherein the system is configured to access mapping data providing a model of the environment and to track a location of the mobile reality
capture device within the model of the environment, wherein the location of the mobile reality capture device is taken into account for deriving a motion category of the determined motion pattern and/or the derivation of the expected motion pattern, (see abstract and claims 1-10 where the neural network can store a database of shapes for comparing the mobile device and the orientation to adjust the camera based on the neural network outputs; and see col. 6, line 10 to col. 7, line 55 where the device can determine that the object is a face or a jeep and then adjust the parameters of the camera accordingly and see col. 8, lines 1-55 where the camera can determine that the person is static based on the feature recognition and then adjust a brightness or that the moving vehicle is detected and then can pan the image as desired; Such an effective extractor would not require additional image processing for image centering, orientation correction, and scale adjustment.)
It would have been obvious for one of ordinary skill in the art to combine the teachings of SADOVNIK with the disclosure of VOLKENBURG with a reasonable expectation of success since SADOVNIK teaches that an improved camera device and 3d scanner can be coupled to a neural network device. The neural network can classify the object as a vehicle or a human face based on a feature vector comparison. The object or vehicle can be rendered as a feature vector and then uploaded to the Bayesian artificial neural network. Then a classification that this is a face or a vehicle can be made and the camera can be adjusted and portion of the image can be panned, or brighten up to provide an automatic adjustment of the camera. See abstract and col. 6, line 1 to col. 7, line 49 and also claims 1-10.
SADOVNIK et al. teaches “.14. The system according to claim 13, wherein the system comprises an object detection algorithm according to claim 11 and the location of the mobile reality capture device is used to identify the spatial constellation for the recognition of the environment-specific measurement situation. (see abstract and claims 1-10 where the neural network can store a database of shapes for comparing the mobile device and the orientation to adjust the camera based on the neural network outputs; and see col. 6, line 10 to col. 7, line 55 where the device can determine that the object is a face or a jeep and then adjust the parameters of the camera accordingly and see col. 8, lines 1-55 where the camera can determine that the person is static based on the feature recognition and then adjust a brightness or that the moving vehicle is detected and then can pan the image as desired; Such an effective extractor would not require additional image processing for image centering, orientation correction, and scale adjustment and see col. 11-12 and the table where a 3d model can be provided with a range of the object to the sensor or a 2d model can be used also.)
It would have been obvious for one of ordinary skill in the art to combine the teachings of SADOVNIK with the disclosure of VOLKENBURG with a reasonable expectation of success since SADOVNIK teaches that an improved camera device and 3d scanner can be coupled to a neural network device. The neural network can classify the object as a vehicle or a human face based on a feature vector comparison. The object or vehicle can be rendered as a feature vector and then uploaded to the Bayesian artificial neural network. Then a classification that this is a face or a vehicle can be made and the camera can be adjusted and portion of the image can be panned, or brighten up to provide an automatic adjustment of the camera. See abstract and col. 6, line 1 to col. 7, line 49 and also claims 1-10.
SADOVNIK et al. teaches “ 15. The system according to claim 1, wherein the 3D surveying unit is embodied as a laser
scanner configured to carry out, during movement of the mobile reality capture device, a scanning movement of a laser measurement beam relative to two rotation axes in order to provide the generation of the 3D measurement data based thereon. . (see abstract and claims 1-10 where the neural network can store a database of shapes for comparing the mobile device and the orientation to adjust the camera based on the neural network outputs; and see col. 6, line 10 to col. 7, line 55 where the device can determine that the object is a face or a jeep and then adjust the parameters of the camera accordingly and see col. 8, lines 1-55 where the camera can determine that the person is static based on the feature recognition and then adjust a brightness or that the moving vehicle is detected and then can pan the image as desired; Such an effective extractor would not require additional image processing for image centering, orientation correction, and scale adjustment and see col. 11-12 and the table where a 3d model can be provided with a range of the object to the sensor or a 2d model can be used also.)
It would have been obvious for one of ordinary skill in the art to combine the teachings of SADOVNIK with the disclosure of VOLKENBURG with a reasonable expectation of success since SADOVNIK teaches that an improved camera device and 3d scanner can be coupled to a neural network device. The neural network can classify the object as a vehicle or a human face based on a feature vector comparison. The object or vehicle can be rendered as a feature vector and then uploaded to the Bayesian artificial neural network. Then a classification that this is a face or a vehicle can be made and the camera can be adjusted and portion of the image can be panned, or brighten up to provide an automatic adjustment of the camera. See abstract and col. 6, line 1 to col. 7, line 49 and also claims 1-10.
The primary reference discloses “...17. A computer program product comprising program code stored on a non-transitory
machine-readable medium, wherein the program code comprises computer-executable instructions for performing, when executed in a surveying system, the method according to claim 16”. (See page 3-5 and systems reconstruct a 3D scene or object based on analysis of multiple overlapping 2D images. Provided common features are visible and identified in the images and camera calibration parameters are known or determined, it is possible to extract 3D metric scene or object information. In some cases, the cameras are pre- calibrated. In other cases, self-calibration is attempted based on the image matches. Fixed station scanners scan a scene from a fixed location. Typically, fixed station scanners are arranged to scan a modulated laser beam in two dimensions and acquire range information by measuring the phase-shift of the reflected modulated laser light or the time-of-flight of a reflected laser pulse. By panning the scanner through 360°, it is possible to produce a 360° panoramic range map of the scene. To scan a complete scene often requires moving the fixed station scanner to a number of different scanning locations. Depending on the size of scene, scanning time is typically 10-30 minutes. Some fixed station scanners also comprise a digital camera that is arranged to capture colour information for each surface point in the scan of the scene so that dual colour and range images can be generated. Other fixed station scanners incorporate multiple lasers to allow acquisition of colour as well as range information.)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEAN PAUL CASS whose telephone number is (571)270-1934. The examiner can normally be reached Monday to Friday 7 am to 7 pm; Saturday 10 am to 12 noon.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Scott A. Browne can be reached at 571-270-0151. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEAN PAUL CASS/Primary Examiner, Art Unit 3666