DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
IDS filed 12/11/2025, 6/30/2025, 1/13/2025 are being considered by the examiner
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Response to Preliminary Amendment
Applicant's preliminary amendment filed 11/29/2024 has been received and entered into the record. As a result, 3, 8, 9, and 16 have been amended. Therefore, claims 1-16 are presented for examination.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term "means" or "step" or a term used as a substitute for "means" that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term "means" or "step" or the generic placeholder is modified by functional language, typically, but not always linked by the transition word "for" (e.g., "means for") or another linking word or phrase, such as "configured to" or "so that"; and
(C) the term "means" or "step" or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word "means" (or "step") in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word "means" (or "step") in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word "means" (or "step") are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word "means" (or "step") are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word "means," but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Such claim limitation(s) is/are:
"a logic unit that executes", "a rule engine to apply", and "a classifier trained to determine" in claim 1;
"a logic unit that executes", "at least one classifier trained to determine", and "a rule engine to receive" in claim 11;
"a rule engine to apply", and "a classifier trained to determine" in claim 13; and
"at least one classifier trained to determine", and "a rule engine to receive" in claim 14;
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 16 is rejected under 35 U.S.C. § 101 because the applicant has provided evidence that the applicant intends the term "computer program element" to include non-statutory matter. The applicant describes a computer program element as including open ended language and thus it is reasonable to interpret it to include all possible mediums, including non-statutory mediums (see paragraph [0021]). The word "element" is insufficient to convey only statutory embodiments to one of ordinary skill in the art absent an explicit and deliberate limiting definition or clear differentiation between a computer element and transitory media in the disclosure. As such, the claim is drawn to a form of energy. Energy is not one of the four categories of invention and therefore, this claim is not statutory. Energy is not a series of steps or acts and thus is not a process. Energy is not a physical article or object and as such is not a machine or manufacture. Energy is not a combination of substances and therefore not a composition of matter.
The Examiner suggests amending the claim to read as a "non-transitory computer readable storage medium".
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 3-16 are rejected under 35 U.S.C. 103 as being unpatentable over Zavesky et al. [US Pub. 2019/0303837] ("Zavesky") in view of Putman et al. [US Pub. 2022/0043420] ("Putman").
With regard to claim 1, Zavesky teaches an automated equipment installation verification system to automatically verify correctness of installation actions ("a performance of a workflow comprising a plurality of tasks relating to the at least one physical item, tracking a progression of the performance of the workflow, and detecting, from at least a first image of the plurality of images, a deviation from the workflow [abstract]") of an operative installing an item of equipment ("a user 299 may be performing the process 200 to assemble the physical item 295 [par. 0030]" and see [par. 0022] where the item can be various objects subject to various actions), the system comprising:
a logic unit ("the machine learning model may comprise a classifier [par. 0032]") ("such verification may include applying a classifier (e.g., a deep learning, neural network-based classifier, or 'deep neural network') to determine whether image 231 'matches' image 221 or not [par. 0033]") according to a sequence of stepwise actions to be performed by the operative in the installation of the equipment ("workflow 210 comprising a plurality of tasks 211-214 (tasks 1-4) for the assembly of physical item 295 [par. 0029]")
each measure of correctness being determined by a classifier trained to determine a degree of correctness of a respective act based on sensor data corresponding to the act ("such verification may include applying a classifier (e.g., a deep learning, neural network-based classifier, or 'deep neural network') to determine whether image 231 'matches' image 221 or not … if the output indicates a likelihood of a match is greater than 50%, greater than 75%, etc., then it may be considered a match in accordance with the example process 200. Thus, the workflow 210 may advance to the second task 212 [par. 0033]").
Although Zavesky teaches a machine learning model, Zavesky does not explicitly teach a rule engine to determine a degree of correctness of an installation of the equipment.
In an analogous art (process verification), Putman teaches a rule engine ("Machine learning classification models [par. 0029]") to determine a degree of correctness of an installation of an equipment ("After classifications/differences have been determined, process 200 can proceed to step 230 in which an analysis of the entire process/workflow is performed, e.g., based on the classifications/differences for each station/node determined in steps 226 and 228 [par. 0063]" and "by accurately quantifying and tracking error contributions from specific segments in an assembly workflow, products can be graded and classified by product quality or deviation quantity. As such, products of certain quality classifications can be steered to different manufacturing processes, or to different customers, i.e., depending on product quality [par. 0026]" and "parts/components can be classified into different quality tiers and/or may be identified for removal or repair, depending on their associated classifications/differences [par. 0062]").
Because Putman teaches determining the degree of product quality based on the culmination of the entire workflow to classify different quality tiers to identify the product for removal or repair [pars. 0026 and 0062], it would have been obvious to one of ordinary skill in the art at the time of filing the invention to have included Putman's teachings, with the teachings of Zavesky, for the benefit of identifying whether the product needs to be removed or repaired.
With regard to claim 3, the combination above teaches the system of claim 1. Zavesky in the combination further teaches wherein the rule engine further identifies at least one act of the operative for which a measure of correctness of the act is below a threshold measure of correctness ("such verification may include applying a classifier (e.g., a deep learning, neural network-based classifier, or 'deep neural network') to determine whether image 231 'matches' image 221 or not … if the output indicates a likelihood of a match is greater than 50%, greater than 75%, etc., then it may be considered a match in accordance with the example process 200. Thus, the workflow 210 may advance to the second task 212 [par. 0033]").
With regard to claim 4, the combination above teaches the system of claim 3. Zavesky in the combination teaches the system further comprising a communications interface, wherein the identified at least one act of the operative for which a measure of correctness of the act is below a threshold measure of correctness is communicated via the communications interface to inform the operative ("the deviation may be detected by device 290 comparing image 232 to image 222 and/or applying a classifier associated with task 212 to image 232 and determining that there is not a match. In one example, device 290 may then notify the user of the deviation, e.g., via audio and/or textual warning, via a reminder by again presenting image 222 for reference via a display of device 290, and so forth [par. 0035]").
With regard to claim 5, the combination above teaches the system of claim 1. Zavesky in the combination teaches the system further comprising a communications interface, wherein the degree of correctness of the installation of the equipment is communicated via the communications interface to inform the operative ("the deviation may be detected by device 290 comparing image 232 to image 222 and/or applying a classifier associated with task 212 to image 232 and determining that there is not a match. In one example, device 290 may then notify the user of the deviation, e.g., via audio and/or textual warning, via a reminder by again presenting image 222 for reference via a display of device 290, and so forth [par. 0035]").
With regard to claim 6, the combination above teaches the system of claim 3. Zavesky in the combination further teaches wherein the logic unit further compares sensor data for each of the at least one act of the operative for which the measure of correctness of the act is below the threshold measure of correctness, and a predefined model act of the operative, the comparison identifying differences therebetween ("such verification may include applying a classifier (e.g., a deep learning, neural network-based classifier, or 'deep neural network') to determine whether image 231 'matches' image 221 or not … if the output indicates a likelihood of a match is greater than 50%, greater than 75%, etc., then it may be considered a match in accordance with the example process 200. Thus, the workflow 210 may advance to the second task 212 [par. 0033]").
With regard to claim 7, the combination above teaches the system of claim 6. Zavesky in the combination teaches the system further comprising a communications interface, wherein identified differences are communicated via the communications interface ("the deviation may be detected by device 290 comparing image 232 to image 222 and/or applying a classifier associated with task 212 to image 232 and determining that there is not a match. In one example, device 290 may then notify the user of the deviation, e.g., via audio and/or textual warning, via a reminder by again presenting image 222 for reference via a display of device 290, and so forth [par. 0035]").
With regard to claim 8, the combination above teaches the system of claim 1. Zavesky in the combination further teaches wherein the sensor data further includes information about one or more of the physical acts including one or more of:
a duration of the act; and
a condition of an environment in which the act was performed ("additional information may be used to verify task completion, such as sensor data gathered by device 290, data gathered by physical item 295 (e.g., if physical item 295 is equipped for wired and/or wireless communication and further includes a sensor for measuring aspects of an environment or an internal monitoring unit for monitoring parameters of the physical item 295, such as a memory utilization, a processor utilization, etc.), sensor data gathered by an independent sensor device, and so forth. In general, the types of data used to confirm task completion may vary depending upon the type of physical object, the capabilities of the device of the user, the types of media available in connection with a workflow, the type of task, and so on [par. 0039]").
Note: claim is presented in the alternative.
With regard to claim 9, the combination above teaches the system of claim 1. Zavesky in the combination further teaches wherein the sensor data includes one or more of:
image sensor data ("such verification may include applying a classifier (e.g., a deep learning, neural network-based classifier, or 'deep neural network') to determine whether image 231 'matches' image 221 or not [par. 0033]");
sound sensor data; and
sensor data about a state of the equipment.
Note: claim is presented in the alternative.
With regard to claim 10, the combination above teaches the system of claim 9. Zavesky in the combination further teaches wherein the degree of correctness of an act is further determined based on sensor data about the state of the equipment ("verifying from image 231 that the user has physically arranged parts A, B, and C as indicated in image 221 … if the output indicates a likelihood of a match is greater than 50%, greater than 75%, etc., then it may be considered a match in accordance with the example process 200. Thus, the workflow 210 may advance to the second task 212 [par. 0033]").
With regard to claim 11, Zavesky teaches an automated equipment installation verification system to automatically verify correctness of installation actions ("a performance of a workflow comprising a plurality of tasks relating to the at least one physical item, tracking a progression of the performance of the workflow, and detecting, from at least a first image of the plurality of images, a deviation from the workflow [abstract]") of an operative installing an item of equipment ("a user 299 may be performing the process 200 to assemble the physical item 295 [par. 0030]" and see [par. 0022] where the item can be various objects subject to various actions), the system comprising:
at least one sensor that senses physical acts performed by the operative installing the equipment and generates data corresponding to each act ("The user 299 may capture an image 231 via device 290 to confirm performance and/or completion of the first task 211 [par. 0030]" and "for images, the relevant information may include low-level features such as colors, color distribution, standard deviation of pixel intensities, contrast, average brightness, shape/edge positions, and so forth, as well high-level semantic information such as what type of object is in the field of view of the camera, what defects are present, etc [par. 0031]");
a logic unit that executes at least one classifier trained to determine a degree of correctness of an act of the operative based on the data corresponding to the act ("such verification may include applying a classifier (e.g., a deep learning, neural network-based classifier, or 'deep neural network') to determine whether image 231 'matches' image 221 or not … if the output indicates a likelihood of a match is greater than 50%, greater than 75%, etc., then it may be considered a match in accordance with the example process 200. Thus, the workflow 210 may advance to the second task 212 [par. 0033]"), the act corresponding to the action in a sequence of stepwise actions ("workflow 210 comprising a plurality of tasks 211-214 (tasks 1-4) for the assembly of physical item 295 [par. 0029]");
wherein the logic unit further communicates an output of the classifier ("such verification may include applying a classifier (e.g., a deep learning, neural network-based classifier, or 'deep neural network') to determine whether image 231 'matches' image 221 or not [par. 0033]").
Although Zavesky teaches a machine learning model, Zavesky does not explicitly teach a rule engine to receive an indication of a degree of correctness of a sequence of acts performed by the operative.
In an analogous art (process verification), Putman teaches a rule engine ("Machine learning classification models [par. 0029]") to receive an indication of a degree of correctness of a sequence of acts performed by the operative ("After classifications/differences have been determined, process 200 can proceed to step 230 in which an analysis of the entire process/workflow is performed, e.g., based on the classifications/differences for each station/node determined in steps 226 and 228 [par. 0063]" and "by accurately quantifying and tracking error contributions from specific segments in an assembly workflow, products can be graded and classified by product quality or deviation quantity. As such, products of certain quality classifications can be steered to different manufacturing processes, or to different customers, i.e., depending on product quality [par. 0026]" and "parts/components can be classified into different quality tiers and/or may be identified for removal or repair, depending on their associated classifications/differences [par. 0062]").
Because Putman teaches determining the degree of product quality based on the culmination of the entire workflow to classify different quality tiers to identify the product for removal or repair [pars. 0026 and 0062], it would have been obvious to one of ordinary skill in the art at the time of filing the invention to have included Putman's teachings, with the teachings of Zavesky, for the benefit of identifying whether the product needs to be removed or repaired.
Additionally, Zavesky teaches, "any data, records, fields, and/or intermediate results discussed in the method can be stored, displayed and/or outputted to another device as required for a particular application [par. 0056]."
It would have been obvious to one of ordinary skill in the art at the time of filing the invention to have output the results of the classifier to the rule engine, in order to allow the rule engine to determine whether the product needs to be removed or repaired.
With regard to claim 12, the combination above teaches the system of claim 11. Zavesky in the combination teaches the system further comprising a data store that stores a digital representation of an action in a sequence of stepwise actions to be performed by the operative in the installation of the equipment ("FIG. 2 illustrates workflow 210 comprising a plurality of tasks 211-214 (tasks 1-4) for the assembly of physical item 295. In addition, in the example of FIG. 2, the tasks 211-214 of workflow 210 may have a plurality of associated media, e.g., images 221-224, showing the physical item 295 at various points during the assembly [par. 0029]"),
wherein the logic unit further compares the data corresponding to an act of the operative with a digital representation of a corresponding action in the data store to identify a difference therebetween ("such verification may include applying a classifier (e.g., a deep learning, neural network-based classifier, or 'deep neural network') to determine whether image 231 'matches' image 221 or not … if the output indicates a likelihood of a match is greater than 50%, greater than 75%, etc., then it may be considered a match in accordance with the example process 200. Thus, the workflow 210 may advance to the second task 212 [par. 0033]").
With regard to claim 13, the combination above teaches claim 1. Claim 13 recites limitations having the same scope as those pertaining to claim 1; therefore, claim 13 is rejected along the same grounds as claim 1.
With regard to claim 14, the combination above teaches claim 11. Claim 14 recites limitations having the same scope as those pertaining to claim 11; therefore, claim 14 is rejected along the same grounds as claim 11.
With regard to claim 15, the combination above teaches claim 12. Claim 15 recites limitations having the same scope as those pertaining to claim 12; therefore, claim 15 is rejected along the same grounds as claim 12.
With regard to claim 16, Zavesky in the combination above teaches a computer program element comprising computer program code to, when loaded into a computer system and executed thereon, cause the computer to perform the steps of a method as claimed in claim 13 ("The processor executing the computer readable or software instructions relating to the above described method(s) can be perceived as a programmed processor or a specialized processor [par. 0060]").
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Zavesky in view of Putman further in view of Suzuki et al. [US Pub. 2004/0083019] ("Suzuki").
With regard to claim 2, the combination of Zavesky and Putman teaches the system of claim 1. Putman in the combination further teaches wherein the logic unit further evaluates
the degree of correctness of the installation of the equipment ("After classifications/differences have been determined, process 200 can proceed to step 230 in which an analysis of the entire process/workflow is performed, e.g., based on the classifications/differences for each station/node determined in steps 226 and 228 [par. 0063]" and "by accurately quantifying and tracking error contributions from specific segments in an assembly workflow, products can be graded and classified by product quality or deviation quantity. As such, products of certain quality classifications can be steered to different manufacturing processes, or to different customers, i.e., depending on product quality [par. 0026]" and "parts/components can be classified into different quality tiers and/or may be identified for removal or repair, depending on their associated classifications/differences [par. 0062]"); and
the measure of correctness for each of the plurality of physical acts.
Note: claim is presented in the alternative.
The combination does not explicitly teach to evaluate a measure of a statistical likelihood that the equipment will fail.
In the same field of endeavor (estimating failure), Suzuki teaches to evaluate a measure of a statistical likelihood that an equipment will fail ("a process or work for assembling a product or an assembled part or subassembly is represented by a combination of predefined standard attaching operations, to thereby calculate or compute a likelihood of occurrence of assembling failure (i.e., assembling-related fraction defective) by totalizing fraction defective coefficients relevant or relating to the individual standard attaching operations, respectively [par. 0054]").
Suzuki further teaches, "Needless to say, any attaching operation is accompanied with the possibility or potential of the assembling-related failure taking place. (Such potential is referred to as the assembling-related fraction defective coefficient.) In this conjunction, it is noted that a major factor which affects primarily the defect occurrence likelihood can be found in the attaching operation [par. 0057]."
Because Putman teaches determining the degree of product quality based on the culmination of the entire workflow to classify different quality tiers [pars. 0026 and 0062], it would have been obvious to one of ordinary skill in the art at the time of filing the invention to have utilized Suzuki's concept of totalizing fraction defective coefficients, with the teachings of Zavesky and Putman, for the benefit of determining whether a product might fail based on the culmination of the workflow.
Citation of Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Sachdeva et al. [US Pub 2022/0101016] teaches receiving, using one or more processors, image data, the image data including a first training video representing performance of one or more steps on a first workpiece; applying, using the one or more processors, a first set of labels to the first training video based on user input; performing, using the one or more processors, extraction on the image data, thereby generating extracted information, the extracted information including first extracted image information associated with the first training video; and training, using the one or more processors, a process monitoring algorithm based on the extracted information and the first set of labels.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINCENT W CHANG whose telephone number is (571)270-1214. The examiner can normally be reached (M-F) 10:00 am - 6:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mohammad Ali can be reached at 571-272-4105. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
VINCENT WEN-LIANG CHANG
Examiner
Art Unit 2119
/MOHAMMAD ALI/Supervisory Patent Examiner, Art Unit 2119
Prosecution Insights