DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Interpretation
Claim 2 recites a limitation of the form “at least one of A and B”. Similarly, Claim 5 recites a limitation of the form “at least one of A, B, C, D, E and F”, claim 14 recites a limitation of the form “at least one of A, B and C” and claim 18 recites a limitation of the form “at least one of A, B, C and D”. In accordance with the U.S. Court of Appeals for the Federal Circuit in SuperGuide Corp v. DirecTV Enterprises, Inc., these limitations are conjunctive in nature and to be construed as “at least one of A, at least one of B, (at least one of C, at least one of D, at least one of E and at least one of F)”. Therefore, these claims are addressed herein as requiring each of these steps rather than the alternative of A or B (or C or D or E or F).
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “an image preprocessing unit configured to extract”, “image analysis unit configured to analyze”, “object detection unit configured to detect”, “object classification unit configured to determine”, “object tracking unit configured to track”, “image sampling unit configured to extract”, “image scaling unit configured to convert”, “image format unit configured to convert”, “risk determination unit configured to perform” in claims 1, 6, 8, 9, 13 and 16.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 6, 8, 9, 12 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Glaser et al (US 2018/0012080).
Regarding claim 1, Glaser et al. discloses a semiconductor fabrication facility analysis system comprising:
one or more surveillance cameras (“As a related application, the system and method of an alternative embodiment may be implemented as capabilities of an imaging device or application. A digital camera, personal computing device with a camera, a wearable computer (e.g., smart glasses), a camera application, a surveillance camera, a robot, an automobile, and/or other suitable devices may leverage the system and method to enable device interface extraction for various applications” at paragraph 0066) configured to generate imaging data comprising captured imagery of a surveillance area within a semiconductor fabrication facility (“The system and method could additionally be applied to industrial use cases that have large, complex integrated systems. Manufacturing plants, power plants, refineries, utility treatment plants, industrial agriculture, chemical laboratory, a fabrication facility, and other industrial settings often have a variety of meters and sensor systems displaying information distributed at different points in the environment” at paragraph 0064, line 1; though not explicitly a semiconductor fabrication facility, the umbrella of fabrication facility would therefore include those for semiconductors);
an image preprocessing unit configured to extract an image from the imaging data (“Block S132, which includes formatting the image data, functions to prepare, normalize, and/or rectify the image data as part of the visual formatting stage. Preferably, formatting of the image data includes applying image data transformations to a region of image data and thereby generating formatted image data of the device interface source. In some variations, the formatted image data prepares the image data to be used in subsequent stages of processing or for a rendered interface representation” at paragraph 0164, line 1); and
an image analysis unit configured to analyze the image (“The computer-executable component can be a processor but any suitable hardware device can (alternatively or additionally) execute the instructions” at paragraph 0255, last sentence), wherein the image analysis unit comprises:
an object detection unit configured to detect a mobile object in the image (“As another potential benefit, the system and method can leverage the collection of extracted data with the collection of image data. The image data is primarily used in extracting interface data from a device, but the image data could additionally be used in detecting the environmental context, which can be used to augment or enhance the interpretation of the extracted data. For example, a medical monitoring device may sense various biometric signals, but the system and method could additionally collect the state of the patient as part of the environmental context. In an industrial application, operating conditions of a machine may be associated with detected human workers and/or activity of those workers” at paragraph 0058, line 1); and
an object classification unit configured to determine an object classification for the mobile object (“With respect to device state detection, the method may additionally include detecting a user-object in the image data and tracking device state change in association with the user-object. The user-object is preferably a CV-detected person. In some cases, the user-object may be limited to specific human identities or classifications of humans (e.g., an worker, patient, customer, doctor, nurse, child, adult, etc.)” at paragraph 0191, line 1) based on a color analysis of the mobile object (“Presence extraction can additionally be used with other forms of physical state detection such that color profiles, size, shape, and/or other detectable attributes can be collected in addition to or in response to detection of some object presence” at paragraph 0135, last sentence; “Parameterizing the visual physical state can include calculating size, tracking shape, tracking color profile, tracking orientation, tracking position within a region, and/or tracking other attributes of the physical state of an object” at paragraph 0178, line 5).
Regarding claim 6, Glaser et al. discloses a system wherein the image analysis unit further comprises an object tracking unit configured to track a movement path of the mobile object (“Parameterizing the visual physical state can include calculating size, tracking shape, tracking color profile, tracking orientation, tracking position within a region, and/or tracking other attributes of the physical state of an object” at paragraph 0178, line 5).
Regarding claim 8, Glaser et al. discloses a semiconductor fabrication facility analysis system comprising:
one or more surveillance cameras (“As a related application, the system and method of an alternative embodiment may be implemented as capabilities of an imaging device or application. A digital camera, personal computing device with a camera, a wearable computer (e.g., smart glasses), a camera application, a surveillance camera, a robot, an automobile, and/or other suitable devices may leverage the system and method to enable device interface extraction for various applications” at paragraph 0066) configured to generate imaging data comprising captured imagery of a surveillance area within a semiconductor fabrication facility (“The system and method could additionally be applied to industrial use cases that have large, complex integrated systems. Manufacturing plants, power plants, refineries, utility treatment plants, industrial agriculture, chemical laboratory, a fabrication facility, and other industrial settings often have a variety of meters and sensor systems displaying information distributed at different points in the environment” at paragraph 0064, line 1; though not explicitly a semiconductor fabrication facility, the umbrella of fabrication facility would therefore include those for semiconductors);
an image preprocessing unit configured to extract an image from the imaging data and preprocess the image (“Block S132, which includes formatting the image data, functions to prepare, normalize, and/or rectify the image data as part of the visual formatting stage. Preferably, formatting of the image data includes applying image data transformations to a region of image data and thereby generating formatted image data of the device interface source. In some variations, the formatted image data prepares the image data to be used in subsequent stages of processing or for a rendered interface representation” at paragraph 0164, line 1); and
an image analysis unit configured to analyze the preprocessed image (“The computer-executable component can be a processor but any suitable hardware device can (alternatively or additionally) execute the instructions” at paragraph 0255, last sentence), wherein the image analysis unit comprises:
an object detection unit configured to detect a mobile object in the preprocessed image (“As another potential benefit, the system and method can leverage the collection of extracted data with the collection of image data. The image data is primarily used in extracting interface data from a device, but the image data could additionally be used in detecting the environmental context, which can be used to augment or enhance the interpretation of the extracted data. For example, a medical monitoring device may sense various biometric signals, but the system and method could additionally collect the state of the patient as part of the environmental context. In an industrial application, operating conditions of a machine may be associated with detected human workers and/or activity of those workers” at paragraph 0058, line 1); and
an object classification unit configured to determine an object classification for the mobile object (“With respect to device state detection, the method may additionally include detecting a user-object in the image data and tracking device state change in association with the user-object. The user-object is preferably a CV-detected person. In some cases, the user-object may be limited to specific human identities or classifications of humans (e.g., an worker, patient, customer, doctor, nurse, child, adult, etc.)” at paragraph 0191, line 1) based on a color analysis of the mobile object (“Presence extraction can additionally be used with other forms of physical state detection such that color profiles, size, shape, and/or other detectable attributes can be collected in addition to or in response to detection of some object presence” at paragraph 0135, last sentence; “Parameterizing the visual physical state can include calculating size, tracking shape, tracking color profile, tracking orientation, tracking position within a region, and/or tracking other attributes of the physical state of an object” at paragraph 0178, line 5).
Regarding claim 9, Glaser et al. discloses a system wherein the image preprocessing unit comprises:
an image sampling unit configured to extract the image by sampling the imaging data (“Additionally, the temporal properties of detection and processing such as sampling frequency may be set” at paragraph 0126, fourth to last sentence);
an image scaling unit configured to convert a size of the image (“For example, screens viewed off angle are transformed to remove key-stoning in order to produce regular rectangular images with aspect ratios that might match that of the screen of the original device” at paragraph 0166, line 6); and
an image format unit configured to convert a format of the image (“Other visual transformations can include imaging transformations, which functions to adjust the color space of the image data for enhanced legibility and/or processing. As with other image data transformations, imaging transformations can be customized to different regions of the image data. Imaging transformations can include adjusting the color space, brightness, contrast level, saturation level, hue, sharpness, white point, black point, and/or altering any suitable imaging variable” at paragraph 0167, line 1).
Regarding claim 12, Glaser et al. discloses a system wherein the mobile object comprises a human (“With respect to device state detection, the method may additionally include detecting a user-object in the image data and tracking device state change in association with the user-object. The user-object is preferably a CV-detected person. In some cases, the user-object may be limited to specific human identities or classifications of humans (e.g., an worker, patient, customer, doctor, nurse, child, adult, etc.)” at paragraph 0191, line 1).
Regarding claim 15, Glaser et al. discloses a system wherein the image analysis unit is configured to analyze the preprocessed image by using a machine learning model (“Various techniques may be employed in object detection and classification of a device interface source such as a “bag of features” approach, convolutional neural networks (CNN), statistical machine learning, or other suitable approaches. Neural networks or CNNS such as Fast regional-CNN (r-CNN), Faster R-CNN, Mask R-CNN, and/or other neural network variations and implementations can be executed as computer vision driven object classification processes” at paragraph 0110, line 1).
Claim(s) 2 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Glaser et al. and Doretto et al. (US 8,165,397).
Regarding claim 2, Glaser et al. discloses a system as described in claim 1 above.
Glaser et al. does not explicitly disclose that the object classification unit is configured to determine the object classification based on at least one of a hue, saturation, and value (HSV) representation of a color of the mobile object and an L*a*b* representation of the color of the mobile object.
Doretto et al. teaches a system in the same field of endeavor of person and object detection and classification, wherein the object classification unit is configured to determine the object classification based on at least one of a hue, saturation, and value (HSV) representation of a color of the mobile object and an L*a*b* representation of the color of the mobile object (“The data conversion device 218 can convert RGB pixel data in the normalized image into other color space data, such as Log-RGB color space data, Lab color space data, HSV data, or YIQ data to create a translated image. For example, the data conversion device 218 may convert the RGB pixel data for an image into Log-RGB data for subsequent processing in appearance labeling device 220. Additionally, the data conversion device 218 may convert the pixel data into Lab color space data for subsequent processing in shape labeling device 230” at col. 5, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize HSV and LAB space conversion as taught by Doretto et al. in the system of Glaser et al. to adjust the legibility of the image data for particular applications (see Glaser et al. in conjunction with claim 9 above).
Regarding claim 7, Glaser et al. discloses a system wherein the mobile object is a human (“With respect to device state detection, the method may additionally include detecting a user-object in the image data and tracking device state change in association with the user-object. The user-object is preferably a CV-detected person. In some cases, the user-object may be limited to specific human identities or classifications of humans (e.g., an worker, patient, customer, doctor, nurse, child, adult, etc.)” at paragraph 0191, line 1).
Glaser et al. does not explicitly disclose that the object classification unit is configured to classify the human as a type of worker based on a color of clothing of the human.
Doretto et al. teaches a system in the same field of endeavor of person and object detection and classification, wherein the object classification unit is configured to classify the human as a particular person based on a color of clothing of the human (“The use of L-shaped regions to calculate a spatial context enables the calculation of a descriptor that embeds the orientation of the count of appearance labels. Thus, in embodiments using an L-shaped region or similar orientation-dependent shape, the morphology of the mask makes the appearance context non-rotation invariant. This can be desirable in embodiments where the image processing device is configured to calculate descriptors identifying clothing appearance because the calculated descriptor readily distinguishes between a person wearing a white T-shirt and black pants, and a second person wearing a black T-shirt and white pants” at col. 12, line 33).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to generate a clothing descriptor as taught by Doretto et al. in the classification of Glaser et al. as the “descriptors calculated using the techniques and principles described above can be distinctive and robust to occlusions, illumination and viewpoint variations” (Doretto et al. at col. 15, line 55) and “enables a system that accurately identifies people for same-day tracking based only on the appearance of their clothing” (col. 15, line 58).
Claim(s) 3 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Glaser et al. and Kumar et al. (US 2024/0057892).
Glaser et al. discloses a system as described in claim 1 above.
Glaser et al. does not explicitly disclose that the object detection unit is further configured to generate a heatmap that shows a location of the mobile object within the semiconductor fabrication facility.
Kumar et al. teaches a system in the same field of endeavor of mobile object tracking, wherein the object detection unit is further configured to generate a heatmap that shows a location of the mobile object within the monitored space (“FIG. 3 is a flowchart illustrating an example process 300 that may be performed by the point tracker component 110 for tracking subject body parts in the video data 104, according to embodiments of the present disclosure. At a step 302, the point tracker component 110 may process the video data 104 using a machine learning model(s) to locate subject body part(s). At a step 304, the point tracker component 110 may generate a heatmap(s) for the subject body part(s) based on processing the video data 104 using the machine learning model(s)” at paragraph 0045, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to generate a heatmap as taught by Kumar et al. using the tracked position data of Glaser et al. to be able to localize each part of the tracked person for accurate pose characterization (see Kumar et al. at paragraphs 0045-0047).
Claim(s) 4 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Glaser et al. and Baek et al. (US 2020/0327465).
Glaser et al. discloses a system as described in claim 1 above.
Glaser et al. does not explicitly disclose that the object detection unit is further configured to generate a graph that shows a movement amount of the mobile object over time.
Baek et al. teaches a system in the same field of endeavor of workplace worker safety monitoring and tracking wherein the object detection unit is further configured to generate a graph that shows a movement amount of the mobile object over time (“Waveform graphs, as shown in FIGS. 9, 10, and 11 may be created and visually displayed for selected joints or markers” at paragraph 0104, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to generate the joint graphs as taught by Baek et al. in the system of Glaser et al. to “determine when workers are being exposed to higher risks” (Baek et al. at paragraph 0104, last sentence).
Claim(s) 5 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Glaser et al., Baek et al., Fu et al (US 2024/0161505) and Amin et al. (US 2022/0027664).
Glaser et al. discloses a system wherein the object detection unit is configured to detect the mobile object based on a machine learning model, and the machine learning model comprises at least one of a fast/faster region convolution neural network (R-CNN) (“Various techniques may be employed in object detection and classification of a device interface source such as a “bag of features” approach, convolutional neural networks (CNN), statistical machine learning, or other suitable approaches. Neural networks or CNNS such as Fast regional-CNN (r-CNN), Faster R-CNN, Mask R-CNN, and/or other neural network variations and implementations can be executed as computer vision driven object classification processes” at paragraph 0110, line 1).
Glaser et al. does not explicitly disclose that, and the machine learning model comprises at least one of a support vector machine (SVM), a principal component analysis (PCA), a region-based fully convolution network (RFCN), a single shot multibox (SSD), and you only look once (YOLO).
Baek et al. teaches a system in the same field of endeavor of workplace worker safety monitoring and tracking wherein the object detection unit is configured to detect the mobile object based on a machine learning model, and the machine learning model comprises at least one of a support vector machine (SVM) and a principal component analysis (“The machine learning module 102 may employ one or more machine learning algorithms such as, but not limited to, a nearest neighbor (NN) algorithm (e.g., k-NN models, replicator NN models, etc.); statistical algorithm (e.g., Bayesian networks, etc.); clustering algorithm (e.g., k-means, mean-shift, etc.); neural networks (e.g., reservoir networks, artificial neural networks, etc.); support vector machines (SVMs); logistic or other regression algorithms; Markov models or chains; principal component analysis (PCA)” at paragraph 0048, line 1).
Fu et al. teaches a system in the same field of endeavor of mobile object tracking wherein the object detection unit is configured to detect the mobile object based on a machine learning model, and the machine learning model comprises at least one of a single shot multibox, and you only look once (“Object detection methods are generally classified as neural network-based methods or non-neural methods. For non-neural methods, it is necessary to first define features using one of the following methods, and then use techniques such as support vector machines (SVM) for classification. Neural techniques, on the other hand, enable object detection without well-defined features and are usually based on Convolutional Neural Networks (CNNs). Non-neural methods, such as Viola-Jones object detection framework based on Haar features, scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG) features, etc., but not limited thereto; neural methods, such as Region Proposals (R-CNN), Single Shot Multibox Detector (SSD), You Only Look Once (YOLO)” at paragraph 0059, line 12).
Amin et al. teaches system in the same field of endeavor of mobile object tracking wherein the object detection unit is configured to detect the mobile object based on a machine learning model, and the machine learning model comprises at least one of a region-based fully convolution network (“An object within an image was detected by the module 7. Detectors of this type operate either as “region proposal” detectors or as individual detectors. In the case of region proposal-based detectors, the frame is divided into different regions and the latter are correspondingly processed by different methods. For this purpose, a so-called RPN network is provided, which generates a set of object proposals (the latter are often equated with the regions), which are forwarded to a neural network for classification and regression. “Fast R-CNN”, “faster, R-CNN” and “RFCN” methods or else generally ROI align methods are used for generating object proposals” at paragraph 0060, line 1).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize any of the machine learning architectures as taught by Baek et al., Fu et al. and Amin et al. for the machine learning of Glaser et al. as alternative ways to detect the mobile objects in the image data.
Claim(s) 10 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Glaser et al. and Gaikwad et al. (US 8,300,890).
Glaser et al. discloses a system as described in claim 1 above.
Glaser et al. does not explicitly disclose that the image sampling unit is configured to sample the imaging data at a sampling rate of 3 frame-per-second (FPS) or more.
Gaikwad et al. teaches a system in the same field of endeavor of person and object tracking, wherein the image sampling unit is configured to sample the imaging data at a sampling rate of 3 frame-per-second (FPS) or more (“Increasing the sample rate may involve increasing the FPS by adjusting the time between frames and/or introducing extra frames (that are extrapolated between the original frames that were captured” at col. 23, line 34; “The user interface may also allow the user to specify the size of the data input, the resolution at which the data is viewed and/or taken, and/or the sample rate (e.g., the number of frames during a specified time period or the number of frames per second) at which the data is viewed and/or taken” at col. 21, line 26; though not explicitly 3 fps or higher, it is feasible that the user may set the sample rate as claimed).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the user interface as taught by Gaikwad et al. to adjust the sample rate of Glaser et al. to allow the system sensitivity to be adjusted for subsequent detection and analysis.
Claim(s) 11, 13, 14, 16, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Glaser et al. and Schoessler et al. (US 2022/0288781).
Regarding claim 11, Glaser et al. discloses a system as described in claim 1 above.
Glaser et al. does not explicitly disclose that the mobile object comprises mobile equipment.
Schoessler et al. teaches a system in the same field of endeavor of workplace worker safety monitoring and tracking wherein the mobile object comprises mobile equipment (“In particular embodiments, the robotic system 100 may perform object detection (what objects are present in the environment), object segmentation, object localization, and object tracking (where are the objects) based on sensor data captured by the multimodal sensors. The more objects are recognized, the more the robotic system 100 may anticipate the task and thus human action and movement” at paragraph 0035, line 1; “The onboard computing system may track multiple components of a robotic limb, such as joints, end-effectors, grippers, fingers, etc., and adjusts their pose accordingly until a desired pose is reached” at paragraph 0019, second to last sentence).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize a workpiece tracking as taught by Schoessler et al. in the system of Glaser et al. to “prevent human-robot collision and therefore potential injuries” (Schoessler et al. at paragraph 0027, line 5).
Regarding claim 13, Glaser et al. discloses a system as described in claim 8 above.
Glaser et al. does not explicitly disclose that the image analysis unit further comprises a risk determination unit configured to perform a risk assessment based on one or more risk parameters associated with the mobile object, wherein the one or more risk parameters include a posture of the mobile object.
Schoessler et al. teaches a system in the same field of endeavor of workplace worker safety monitoring and tracking wherein the image analysis unit further comprises a risk determination unit configured to perform a risk assessment based on one or more risk parameters associated with the mobile object, wherein the one or more risk parameters include a posture of the mobile object (“In particular embodiments, objects attributes 704, objects poses 706, task information 708, and 3D scene representation 710 may be provided to an object pose and trajectory forecaster 717. The object pose and trajectory forecaster 714 may generate object pose and trajectory forecast 716, which may comprise future pose and trajectory of an object with respect to time t. In particular embodiments, the robotic system 100 may predict, based on the determined pose and attributes of each object and the actions associated with the respective object, a trajectory of the respective object” at paragraph 0041, line 1; “If a potential collision is predicted, the robot system 100 may adjust the current trajectory” at paragraph 0049, third to last sentence).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the collision prediction as taught by Schoessler et al. in the system of Glaser et al. to “prevent human-robot collision and therefore potential injuries” (Schoessler et al. at paragraph 0027, line 5).
Regarding claim 14, the Glaser et al. and Schoessler et al. combination discloses a system wherein the object tracking unit is configured to determine the posture of the mobile object by using at least one of a recurrent neural network, a long short-term memory, and a convolution neural network (“In particular embodiments, the deep learning algorithms 1318 may include any artificial neural networks (ANNs) that may be utilized to learn deep levels of representations and abstractions from large amounts of data. For example, the deep learning algorithms 1318 may include ANNs, such as a multilayer perceptron (MLP), an autoencoder (AE), a convolution neural network (CNN), a recurrent neural network (RNN), long short term memory (LSTM)” Schoessler et al. at paragraph 0069, line 1; “Variations of neural networks utilized for classification may include, for example and not by way of limitation, three-dimensional segmentation networks (3DSNs) such as three-dimensional convolutional neural networks (3DCNNs)” Schoessler et al. at paragraph 0025, third to last sentence).
Regarding claim 16, Glaser et al. discloses a semiconductor fabrication facility analysis system comprising:
one or more surveillance cameras (“As a related application, the system and method of an alternative embodiment may be implemented as capabilities of an imaging device or application. A digital camera, personal computing device with a camera, a wearable computer (e.g., smart glasses), a camera application, a surveillance camera, a robot, an automobile, and/or other suitable devices may leverage the system and method to enable device interface extraction for various applications” at paragraph 0066) configured to generate imaging data comprising captured imagery of a surveillance area within a semiconductor fabrication facility (“The system and method could additionally be applied to industrial use cases that have large, complex integrated systems. Manufacturing plants, power plants, refineries, utility treatment plants, industrial agriculture, chemical laboratory, a fabrication facility, and other industrial settings often have a variety of meters and sensor systems displaying information distributed at different points in the environment” at paragraph 0064, line 1; though not explicitly a semiconductor fabrication facility, the umbrella of fabrication facility would therefore include those for semiconductors);
an image preprocessing unit configured to extract an image from the imaging data and preprocess the image (“Block S132, which includes formatting the image data, functions to prepare, normalize, and/or rectify the image data as part of the visual formatting stage. Preferably, formatting of the image data includes applying image data transformations to a region of image data and thereby generating formatted image data of the device interface source. In some variations, the formatted image data prepares the image data to be used in subsequent stages of processing or for a rendered interface representation” at paragraph 0164, line 1), wherein the image preprocessing unit comprises:
an image sampling unit configured to extract the image by sampling the imaging data (“Additionally, the temporal properties of detection and processing such as sampling frequency may be set” at paragraph 0126, fourth to last sentence);
an image scaling unit configured to convert a size of the image (“For example, screens viewed off angle are transformed to remove key-stoning in order to produce regular rectangular images with aspect ratios that might match that of the screen of the original device” at paragraph 0166, line 6); and
an image format unit configured to convert a format of the image (“Other visual transformations can include imaging transformations, which functions to adjust the color space of the image data for enhanced legibility and/or processing. As with other image data transformations, imaging transformations can be customized to different regions of the image data. Imaging transformations can include adjusting the color space, brightness, contrast level, saturation level, hue, sharpness, white point, black point, and/or altering any suitable imaging variable” at paragraph 0167, line 1); and
an image analysis unit configured to analyze the preprocessed image (“The computer-executable component can be a processor but any suitable hardware device can (alternatively or additionally) execute the instructions” at paragraph 0255, last sentence), wherein the image analysis unit comprises:
an object detection unit configured to detect a mobile object in the preprocessed image (“As another potential benefit, the system and method can leverage the collection of extracted data with the collection of image data. The image data is primarily used in extracting interface data from a device, but the image data could additionally be used in detecting the environmental context, which can be used to augment or enhance the interpretation of the extracted data. For example, a medical monitoring device may sense various biometric signals, but the system and method could additionally collect the state of the patient as part of the environmental context. In an industrial application, operating conditions of a machine may be associated with detected human workers and/or activity of those workers” at paragraph 0058, line 1);
an object tracking unit configured to track a movement path of the mobile object (“Parameterizing the visual physical state can include calculating size, tracking shape, tracking color profile, tracking orientation, tracking position within a region, and/or tracking other attributes of the physical state of an object” at paragraph 0178, line 5);;
an object classification unit configured to determine an object classification for the mobile object (“With respect to device state detection, the method may additionally include detecting a user-object in the image data and tracking device state change in association with the user-object. The user-object is preferably a CV-detected person. In some cases, the user-object may be limited to specific human identities or classifications of humans (e.g., an worker, patient, customer, doctor, nurse, child, adult, etc.)” at paragraph 0191, line 1) based on a color analysis of the mobile object (“Presence extraction can additionally be used with other forms of physical state detection such that color profiles, size, shape, and/or other detectable attributes can be collected in addition to or in response to detection of some object presence” at paragraph 0135, last sentence; “Parameterizing the visual physical state can include calculating size, tracking shape, tracking color profile, tracking orientation, tracking position within a region, and/or tracking other attributes of the physical state of an object” at paragraph 0178, line 5).
Glaser et al. does not explicitly disclose that the image analysis unit comprises a risk determination unit configured to perform a risk assessment based on one or more risk parameters associated with the mobile object, wherein the one or more risk parameters include a posture of the mobile object.
Schoessler et al. teaches a system in the same field of endeavor of workplace worker safety monitoring and tracking wherein the image analysis unit comprises a risk determination unit configured to perform a risk assessment based on one or more risk parameters associated with the mobile object, wherein the one or more risk parameters include a posture of the mobile object (“In particular embodiments, objects attributes 704, objects poses 706, task information 708, and 3D scene representation 710 may be provided to an object pose and trajectory forecaster 717. The object pose and trajectory forecaster 714 may generate object pose and trajectory forecast 716, which may comprise future pose and trajectory of an object with respect to time t. In particular embodiments, the robotic system 100 may predict, based on the determined pose and attributes of each object and the actions associated with the respective object, a trajectory of the respective object” at paragraph 0041, line 1; “If a potential collision is predicted, the robot system 100 may adjust the current trajectory” at paragraph 0049, third to last sentence).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the collision prediction as taught by Schoessler et al. in the system of Glaser et al. to “prevent human-robot collision and therefore potential injuries” (Schoessler et al. at paragraph 0027, line 5).
Regarding claim 19, the Glaser et al. and Schoessler et al. combination discloses a system wherein the object tracking unit is configured to track a state of a joint of the mobile object (“The onboard computing system may track multiple components of a robotic limb, such as joints, end-effectors, grippers, fingers, etc., and adjusts their pose accordingly until a desired pose is reached. A pose may include either of, or both of, the position in three-dimensional (3D) space and the orientation of the one or more components of the robotic limb” Schoessler et al. at paragraph 0019, last sentence).
Regarding claim 20, the Glaser et al. and Schoessler et al. combination discloses a system wherein the object tracking unit is configured to determine a posture of the mobile object based on a posture estimation algorithm, wherein the one or more risk parameters associated with the mobile object include the posture of the mobile object (“In particular embodiments, objects attributes 704, objects poses 706, task information 708, and 3D scene representation 710 may be provided to an object pose and trajectory forecaster 717. The object pose and trajectory forecaster 714 may generate object pose and trajectory forecast 716, which may comprise future pose and trajectory of an object with respect to time t. In particular embodiments, the robotic system 100 may predict, based on the determined pose and attributes of each object and the actions associated with the respective object, a trajectory of the respective object” Schoessler et al. at paragraph 0041, line 1).
Claim(s) 17 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Glaser et al. and Schoessler et al. as applied to claim 16 above, and further in view of Kumar et al.
The Glaser et al. and Schoessler et al. combination discloses the elements of claim 16 as described above.
The Glaser et al. and Schoessler et al. combination does not explicitly disclose that the object tracking unit is configured to track a movement path of the mobile object by extracting a direction vector of the mobile object.
Kumar et al. teaches a system in the same field of endeavor of mobile object tracking, wherein the object tracking unit is configured to track a movement path of the mobile object by extracting a direction vector of the mobile object (“To determine the lateral displacements, the posture analysis component 130 may first, at a step 602, determine using the point data 112, a displacement vector for a stride interval determined to be analyzed at the step 408. The posture analysis component 130 may determine the displacement vector for each stride interval for the time period. In some embodiments, the point data 112 may be for the keypoints representing a spine center of the subject. The stride interval may span over multiple video frames. In some embodiments, the displacement vector may be a vector connecting the spine center in a first video frame of the stride interval and the spine center in the last video frame of the stride interval” at paragraph 0067).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to characterize the object posture as taught by Kumar et al. using the tracked position data of the Glaser et al. and Schoessler et al. combination to be able to localize each part of the tracked person for accurate pose characterization (see Kumar et al. at paragraphs 0045-0047).
Claim(s) 18 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Glaser et al. and Schoessler et al. as applied to claim 16 above, and further in view of Baek et al. with support from Lu et al. (US 2023/0252649).
The Glaser et al. and Schoessler et al. combination discloses the elements of claim 16 as described above.
The Glaser et al. and Schoessler et al. combination does not explicitly disclose that the object tracking unit is configured to track a movement path of the mobile object by using at least one of a Kalman filter, a combinatorial optimization algorithm, a simple online and real-time tracking (SORT), and a DeepSORT model.
Baek et al. teaches a system in the same field of endeavor of workplace worker safety monitoring and tracking wherein the object tracking unit is configured to track a movement path of the mobile object by using at least one of a Kalman filter (“A Kalman filter may be used as a tracking model since it provides an efficient yet reliable tracking performance” at paragraph 0068, line 1), a combinatorial optimization algorithm (“Here, similar to the original Deep SORT algorithm, we use the Hungarian algorithm to solve the optimal assignment problem between trackers and person detection results” at paragraph 0072, line 10; see Lu et al. at paragraph 0238 that explicitly states that the Hungarian algorithm is a combinatorial optimization problem), a simple online and real-time tracking (“A preferred tracking algorithm is a variant of Deep SORT algorithm described as follows. Similar to Deep SORT, our algorithm begins with some trackers initialized at the bounding boxes detected in the first frame of the video” at paragraph 0065, line 1), and a DeepSORT model (“Each of the identified workers 14 within a bounding box 18 is then tracked by the computing device 16 using the algorithm shown illustrated in FIG. 5. The preferred framework for tracking the workers is DeepSORT (a type of Simple Realtime Tracker that relies on deep machine learning)” at paragraph 0057, line 1).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize any of the tracking architectures as taught by Baek et al. for the object tracking of the Glaser et al. and Schoessler et al. combination as alternative ways to track the mobile objects in the image data.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/ Primary Examiner, Art Unit 2672
Prosecution Insights