DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Foreign priority is not filed.
Information Disclosure Statement
IDS has not been filed.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-22 are rejected under 35 U.S.C. 103 as being unpatentable over Miao et al. (US Pub No. 20180352144 A1) in view of Koch et al. (US Pub No. 20150294492 A1).
Regarding Claim 1,
Miao discloses A system for automated motion capture, comprising: a radar gun configured to track speed of a moving object; an embedded computer communicatively coupled to the radar gun and configured to: receive speed data from the radar gun, decode the speed data, and generate trigger signals based on the decoded speed data; (Miao, [0022-0024], discloses methods and systems for multi-target tracking and focusing based on deep machine learning and laser radar. The disclosed methods and systems may be based on digital image processing for tracking and focusing technology, and may be applied to various types of images and imaging systems, such as camera, video recording, etc. Digital image processing based on deep machine learning can effectively recognize multiple targets and accurately track the targets. By further combining accurate distance measurement obtained by laser radar for only the recognized targets rather than the entire scene, costs associated with devices can be reduced and multi-target tracking and focusing can be achieved. Accordingly, conventional problems associated with tracking and focusing on a moving target, such as a low target recognition rate, tracking instability, and focus instability or focus failure, may be solved. As used in the disclosed embodiments, a target is an object being tracked, and a moving target is an object being tracked that is moving relative to an imaging system used for tracking, such that at least one of the imaging system and target object is moving relative to the other; laser radar distance measurement refers to a process for measuring a distance to a target by illuminating that target with a pulsed laser light, and measuring the reflected pulses with a sensor, such as using a light detection and ranging (LiDAR) technique. For example, an infrared laser device may send a laser pulse in a narrow beam towards an object, and a period of time taken by the pulse to be reflected off the object and returned to the sender of the laser pulse is determined. A distance from the object to the laser device can be calculated based on the measured time elapsed between when the pulse was transmitted and when its reflected pulse was received and the speed of light. Although the disclosed embodiments of the present disclosure are described using such a laser radar distance measurement, other suitable distance measurement techniques, such as ultrasonic distance measurement may also be employed; Deep machine learning may refer to a class of machine learning algorithms that may use interconnected “layers” of linear and/or nonlinear processing devices or software, e.g., configured to perform image feature extraction and/or transformation, in which each successive layer uses an output from a previous layer as its input. Deep machine learning may be supervised (e.g., classification) or unsupervised (e.g., pattern analysis), and higher level features may be derived from lower level features to form a hierarchical representation of data (e.g., pixels of an image). An observation (e.g., an image, audio data, etc.) that is processed by deep machine learning may be represented in many ways; for example, a vector of intensity values per pixel, a set of edges, regions of a particular shape, sampled signals, spectral frequencies, etc. Deep machine learning architectures may include deep neural networks, convolutional deep neural networks, deep belief networks, recurrent neural networks, and so forth; signal data is obtained using laser radar and processed to derive speed or motion of vehicle)
a machine learning module configured to optimize camera settings of the one or more machine vision cameras based on analysis of captured video footage. (Miao, [0025-0028], discloses neural network is a computational model or system based on a collection of individual neural units (i.e., neurons). The collection of neural units may be organized into different layers of neural units, for example, an input layer, one or more hidden layers, and an output layer. Each neural unit may be connected with many other neural units of different layers, and be computed using an activation function (e.g., a summation function). Observations (e.g., image data or patterns) may be presented to the neural network via the input layer, which communicates information corresponding to the observation to the one or more hidden layers where the actual processing may be done using an activation function and a model of weighted connections. The hidden layers may link to the output layer, which in turn may provide an output determined from processing the observation. In some embodiments, a neural network may be self-learning and trained by examples. For example, learning rules may be applied to neural networks to modify one or more weights (e.g., scalar parameters) of the connections according to the input observation; a suitable neural network is selected, and the characteristics of corresponding layers in the network are understood. The neural network is preferably trained, e.g., using a large number of samples collected in different environments. The weights obtained from training may be used to recognize targets in the image data that is input to the neural network. Further, by testing the neural network using image of objects in different environments and the weight parameters determined from training the network, a tracking algorithm may be used to track one or more target objects in the image data. In some embodiments, one or more distance measurements by laser radar may be used to measure a distance from an imaging device to only certain target objects that are of interest to a user. Automatic focusing by the imaging device on a desired target object may be achieved by automatically adjusting focus length of the imaging device based on the measured distance; focusing only on a target object of interest, for example selecting a fixed focus window containing the desired target object, the time and computation cost associated with tracking and focusing on the target object can be significantly reduced, and interference resulting from background information can be significantly decreased, enabling real-time target tracking and flexible selection of focus on only one or more desired target objects; the following exemplary embodiments are described in the context of a movable object, such as a UAV, those skilled in the art will appreciate other implementations are possible and alternative embodiments may be deployed without using a UAV. For example, the system and method disclosed herein may be implemented using various imaging systems, for example on moving or stationary objects, or as part of a larger system consistent with the disclosed embodiments; based on the processed motion of signal data obtained by radar data and sensor image data, the camera sensor parameters are adjusted or optimized to focus the specific object (vehicle) in order to track)
Miao does not explicitly disclose one or more machine vision cameras communicatively coupled to the embedded computer and configured to capture high-speed video footage in response to the trigger signals;
Koch discloses one or more machine vision cameras communicatively coupled to the embedded computer and configured to capture high-speed video footage in response to the trigger signals; (Koch, [0065], discloses computer inputs received by the directional controls 524 may be calculated by the computer system using predictive methods to ensure that the subject is always in an optimal view of each camera. In some embodiments, the predictive methods can analyze the speed, position, and/or acceleration of the subject in order to ensure that the cameras move in time to always keep the subject in view. Additionally, the predictive methods can adjust the zoom characteristics of the camera system 500 in order to ensure that the view of the subject in each frame occupies a near constant area. For example, the zoom characteristics could be manipulated such that the subject occupies 40% of the area of each frame regardless of the distance between the subject and each particular camera; computer processors determine the motion speed of the subject (vehicle) and process it to capture vehicle moving with speed by keeping the vehicle in view all time)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Miao in view of Koch having a method of adjusting camera settings based on captured radar signals of the moving vehicle to focus on the tracked vehicle using the radar signals, with the teachings of Koch having, by the module, continuously focusing on the vehicle object to be tracked once the signals are processed and camera is optimized to captured the high speed motion vehicle in order to accurately track vehicle of speed in applications including managing road traffic regulations.
Regarding Claim 2,
The combination of Miao and Koch further discloses wherein the camera settings optimized by the machine learning module include at least one of shutter speed, frame rate, resolution, and light sensitivity (ISO). (Koch, [0040-0041], discloses wide angle cameras can be used to capture the entire motion capture area at once. In other embodiments, at least some of the cameras may be motion controlled such that they zoom, rotate, and redirect their views such that a high-quality (e.g. high resolution, high dynamic range, and low noise) image of the subject is captured throughout the motion sequence. In order to keep each of the cameras focused on the subject throughout the motion sequence, the subject location can be tracked using two or more tracking cameras, RF tags, and/or any other method of tracking an object in real 3-D space; in order to light the subject evenly during the motion sequence, dynamic lighting controls can be used such that light can be reduced as the subject moves closer to a particular light. Any lighting system may be used that can provide even lighting on the subject. For example, ring lighting may be used on each of the cameras to illuminate the subject. As the subject moves closer to a particular camera, the ring lighting may be reduced such that the subject is not overly saturated with light from that camera perspective. Embodiments that do not relight the 3D models may not need to balance the lighting during the capture phase. When the 3-D model undergoes a relighting process, balancing the lighting during the video capture phase can ensure that the illumination of the subject fits within the dynamic range of each camera as closely as possible. For example, as the subject comes closer to a particular light, the illumination of the subject will increase on the side closest to the light. Therefore, the output of this particular light can be adjusted such that the illumination of the subject remains balance and even; settings (parameters) such as light, zoom (resolution) are controlled (optimized) of the camera capturing the subject (vehicle) in motion (speed)). Additionally, the rational and motivation to combine the references Miao and Koch as applied in rejection of claim 1 apply to this claim.
Regarding Claim 3,
The combination of Miao and Koch further discloses wherein the machine learning module employs reinforcement learning techniques to optimize the camera settings. (Miao, [0022], discloses provide methods and systems for multi-target tracking and focusing based on deep machine learning and laser radar. The disclosed methods and systems may be based on digital image processing for tracking and focusing technology, and may be applied to various types of images and imaging systems, such as camera, video recording, etc. Digital image processing based on deep machine learning can effectively recognize multiple targets and accurately track the targets. By further combining accurate distance measurement obtained by laser radar for only the recognized targets rather than the entire scene, costs associated with devices can be reduced and multi-target tracking and focusing can be achieved. Accordingly, conventional problems associated with tracking and focusing on a moving target, such as a low target recognition rate, tracking instability, and focus instability or focus failure, may be solved. As used in the disclosed embodiments, a target is an object being tracked, and a moving target is an object being tracked that is moving relative to an imaging system used for tracking, such that at least one of the imaging system and target object is moving relative to the other; machine learning adapts to the continuous learning and optimizes camera settings according to focus of the targets tracked (reinforcement learning)). Additionally, the rational and motivation to combine the references Miao and Koch as applied in rejection of claim 1 apply to this claim.
Regarding Claim 4,
The combination of Miao and Koch further discloses wherein the embedded computer is further configured to: identify a motion event based on predefined criteria applied to the decoded speed data; and generate the trigger signals in response to identifying the motion event. (Koch, [0065], discloses computer inputs received by the directional controls 524 may be calculated by the computer system using predictive methods to ensure that the subject is always in an optimal view of each camera. In some embodiments, the predictive methods can analyze the speed, position, and/or acceleration of the subject in order to ensure that the cameras move in time to always keep the subject in view. Additionally, the predictive methods can adjust the zoom characteristics of the camera system 500 in order to ensure that the view of the subject in each frame occupies a near constant area. For example, the zoom characteristics could be manipulated such that the subject occupies 40% of the area of each frame regardless of the distance between the subject and each particular camera; speed of vehicle is tracked and based on the speed (motion event), the camera sensor is triggered to change the settings to keep the vehicle in view). Additionally, the rational and motivation to combine the references Miao and Koch as applied in rejection of claim 1 apply to this claim.
Regarding Claim 5,
The combination of Miao and Koch further discloses wherein the radar gun is configured to continuously track the speed and trajectory of the moving object in real-time. (Koch, [0065], discloses computer inputs received by the directional controls 524 may be calculated by the computer system using predictive methods to ensure that the subject is always in an optimal view of each camera. In some embodiments, the predictive methods can analyze the speed, position, and/or acceleration of the subject in order to ensure that the cameras move in time to always keep the subject in view. Additionally, the predictive methods can adjust the zoom characteristics of the camera system 500 in order to ensure that the view of the subject in each frame occupies a near constant area. For example, the zoom characteristics could be manipulated such that the subject occupies 40% of the area of each frame regardless of the distance between the subject and each particular camera; speed of vehicle is tracked and based on the speed (motion event), the camera sensor is triggered to change the settings to keep the vehicle in view). Additionally, the rational and motivation to combine the references Miao and Koch as applied in rejection of claim 1 apply to this claim.
Regarding Claim 6,
The combination of Miao and Koch further discloses wherein the one or more machine vision cameras are configured to capture video at 1000+ frames per second. (Miao, [0068], discloses tracking may be performed only for a target of interest, e.g., to reduce costs and increase the frame rate. In such a scenario, the target of interest may be identified via a neural network of deep machine learning which recognizes the target of interest to be within a certain region of an image, for example, recognizing the dog within a bounding box in FIG. 4A. In such embodiments, the system may reduce computational costs for tracking, focusing on, and imaging the target of interest. Further to these embodiments, when the target of interest is switched, for example, from a first target (e.g., the dog in FIG. 4A) to a second target (e.g., the car in FIG. 4A), tracking of the second target (e.g., the car) can be performed fast and efficiently due to the close proximity of the second target to the first target, thereby not only reducing computational costs associated with tracking, focusing on, and imaging the second target, but also increasing the frame rate of imaging targets; frame rate is optimized to capture range of frames per second). Additionally, the rational and motivation to combine the references Miao and Koch as applied in rejection of claim 1 apply to this claim.
Regarding Claim 7,
The combination of Miao and Koch further discloses a data storage unit configured to store the captured high-speed video footage and extracted biomechanical data. (Miao, [0051], discloses information and data from sensing system 18 may be communicated to and stored in non-transitory computer-readable medium of memory 52. The computer-readable medium associated with memory 52 may also be configured to store logic, code and/or program instructions executable by processor 54 to perform any suitable embodiment of the methods described herein. For example, the computer-readable medium associated with memory 52 may be configured to store computer-readable instructions that, when executed by processor 54, cause the processor to perform a method comprising one or more steps. The method performed by the processor based on the instructions stored in the non-transitory computer readable medium may involve processing inputs, such as inputs of data or information stored in the non-transitory computer-readable medium of memory 52, inputs received from terminal 32, inputs received from sensing system 18 (e.g., received directly from sensing system or retrieved from memory), and/or other inputs received via communication system 20. The non-transitory computer-readable medium may be configured to store sensing data from the sensing module to be processed by the processing unit. In some embodiments, the non-transitory computer-readable medium can be used to store the processing results produced by the processing unit; sensed data is stored in memory storage). Additionally, the rational and motivation to combine the references Miao and Koch as applied in rejection of claim 1 apply to this claim.
Regarding Claim 8,
The combination of Miao and Koch further discloses wherein the machine learning module is further configured to analyze object trajectories across multiple capture sessions, and improve trigger signal timing accuracy based on the analysis. (Miao, [0022], discloses provide methods and systems for multi-target tracking and focusing based on deep machine learning and laser radar. The disclosed methods and systems may be based on digital image processing for tracking and focusing technology, and may be applied to various types of images and imaging systems, such as camera, video recording, etc. Digital image processing based on deep machine learning can effectively recognize multiple targets and accurately track the targets. By further combining accurate distance measurement obtained by laser radar for only the recognized targets rather than the entire scene, costs associated with devices can be reduced and multi-target tracking and focusing can be achieved. Accordingly, conventional problems associated with tracking and focusing on a moving target, such as a low target recognition rate, tracking instability, and focus instability or focus failure, may be solved. As used in the disclosed embodiments, a target is an object being tracked, and a moving target is an object being tracked that is moving relative to an imaging system used for tracking, such that at least one of the imaging system and target object is moving relative to the other; machine learning adapts to the continuous learning and optimizes camera settings according to focus of the targets tracked (reinforcement learning)). Additionally, the rational and motivation to combine the references Miao and Koch as applied in rejection of claim 1 apply to this claim.
Regarding Claim 9,
The combination of Miao and Koch further discloses wherein the embedded computer is further configured to apply a Kalman filter to smooth out noise in the speed data received from the radar gun. (Koch, [0079], discloses in order to generate a 3-D representation of the subject in each frame, the computer system may begin by analyzing the machine vision video sequences. Embodiments such as those described above where the machine vision camera captures the motion of the subject using the IR emitter/lens combination, the image of the subject in each frame may appear speckled with the IR noise pattern such that the background can easily be separated from the geometry of the subject. Fig. 9 illustrates an image captured by the machine vision camera of the subject illustrating the IR pattern and contrasting the image of the subject with the darker background. A frame such as that illustrated by Fig. 9 can be analyzed for each of the video sequences. For example, in one of the embodiments described above, 16 frames from 16 different cameras can be analyzed. Each of the 16 frames can represent the subject from a different perspective at the same moment in time; noise is filtered from the IR data). Additionally, the rational and motivation to combine the references Miao and Koch as applied in rejection of claim 1 apply to this claim.
Regarding Claim 10,
The combination of Miao and Koch further discloses wherein the machine learning module is configured to optimize the camera settings in real-time during a capture session. (Miao, [0022], discloses provide methods and systems for multi-target tracking and focusing based on deep machine learning and laser radar. The disclosed methods and systems may be based on digital image processing for tracking and focusing technology, and may be applied to various types of images and imaging systems, such as camera, video recording, etc. Digital image processing based on deep machine learning can effectively recognize multiple targets and accurately track the targets. By further combining accurate distance measurement obtained by laser radar for only the recognized targets rather than the entire scene, costs associated with devices can be reduced and multi-target tracking and focusing can be achieved. Accordingly, conventional problems associated with tracking and focusing on a moving target, such as a low target recognition rate, tracking instability, and focus instability or focus failure, may be solved. As used in the disclosed embodiments, a target is an object being tracked, and a moving target is an object being tracked that is moving relative to an imaging system used for tracking, such that at least one of the imaging system and target object is moving relative to the other; machine learning adapts to the continuous learning and optimizes camera settings according to focus of the targets tracked (reinforcement learning)). Additionally, the rational and motivation to combine the references Miao and Koch as applied in rejection of claim 1 apply to this claim.
Regarding Claim 11,
The combination of Miao and Koch further discloses wherein the embedded computer is configured to send trigger signals to the one or more machine vision cameras via at least one of a genlock connection and a multicast packet. (Koch, [0097-0098], discloses each of the embodiments disclosed herein may be implemented in a special-purpose computer system. FIG. 16 illustrates an exemplary computer system 1600, in which parts of various embodiments of the present invention may be implemented. The system 1600 may be used to implement any of the computer systems described above. The computer system 1600 is shown comprising hardware elements that may be electrically coupled via a bus 1655. The hardware elements may include one or more central processing units (CPUs) 1605, one or more input devices 1610 (e.g., a mouse, a keyboard, etc.), and one or more output devices 1615 (e.g., a display device, a printer, etc.). The computer system 1600 may also include one or more storage device 1620. By way of example, storage device(s) 1620 may be disk drives, optical storage devices, solid-state storage device such as a random access memory (RAM) and/or a read-only memory (ROM), which can be programmable, flash-updateable and/or the like; computer system 1600 may additionally include a computer-readable storage media reader 1625a, a communications system 1630 (e.g., a modem, a network card (wireless or wired), an infra-red communication device, etc.), and working memory 1640, which may include RAM and ROM devices as described above. In some embodiments, the computer system 1600 may also include a processing acceleration unit 1635, which can include a DSP, a special-purpose processor and/or the like; multiple camera sensor inputs are synched with each other to be processed). Additionally, the rational and motivation to combine the references Miao and Koch as applied in rejection of claim 1 apply to this claim.
Claims 12-19 recite method with steps corresponding to the device elements recited in Claims 1-8. Therefore, the recited steps of the method claims 12-19 are mapped to the proposed combination in the same manner as the corresponding elements of Claims 1-8. Additionally, the rationale and motivation to combine the Miao and Koch references presented in rejection of Claim 1, apply to these claims.
Claims 20-22 recite system with elements corresponding to the device elements recited in Claims 1-3. Therefore, the recited elements of the system claims 20-22 are mapped to the proposed combination in the same manner as the corresponding elements of Claims 1-3. Additionally, the rationale and motivation to combine the Miao and Koch references presented in rejection of Claim 1, apply to these claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US-20210190936-A1 (Sabripour et al., system (100) has a calibrated camera device that is configured to generate image data to track a movement of an identified object during a period of time and to update a track to add first metadata in respect of the identified object, exit position, exit motion vector and object classification data of a first type. A calibrated radar device is configured to carry out a comparison to determine whether the new object is the identified object between first metadata and second metadata. The calibrated radar device is configured to update the track to include both first metadata and second metadata together as transformed hybrid data, when the new object is determined to be the identified object; Abstract).
US-20230245433-A1 (He et al., Systems and methods for implementing a hybrid machine vision model to optimize performance of a machine vision job are disclosed herein. An example method includes: (a) receiving, at a machine vision job including one or more machine vision tools, a set of training images; (b) generating, by the machine vision tools, prediction values corresponding to the set of training images; (c) inputting the prediction values into a machine learning (ML) model configured to receive prediction values and output a change value corresponding to the machine vision job; (d) adjusting the machine vision job based on the change value to improve performance of the machine vision job; (e) iteratively performing steps (a)-(e) until the ML model determines that the prediction values satisfy a prediction threshold; and executing, on a machine vision camera, the machine vision job to analyze a run-time image of a target object and output an inspection result, Abstract)
US-10696272-B2 (Salter et al., A lighting system of a vehicle is provided herein and includes an external lighting device, one or more sensors configured to detect an object approaching the vehicle, and a controller configured to identify the object and assess a threat level thereof based on input from the one or more sensors. If the controller determines that the object is high threat, the controller operates the external lighting device to produce an illumination sequence in the direction of the object, Abstract)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PINALBEN V PATEL whose telephone number is (571)270-5872. The examiner can normally be reached M-F: 10am - 8pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at 571-272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Pinalben Patel/Examiner, Art Unit 2673
Prosecution Insights