DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged that application claims priority to foreign application with application number GB2011194.4 dated 07/20/2020. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Information Disclosure Statement
The IDS dated 01/27/2026 has been considered and placed in the application file.
Response to Arguments
The objections for claims 1-10, 12, 14, 16, 20-22, and 25 have been withdrawn in light of the amended claims.
The Applicants arguments for 35 U.S.C. 112(f) have been considered but are unpersuasive. The “output device” was not amended and “image analysis module” and “validator module” all respectively does not recite sufficient structure in their amended limitations. A neural network-based classifier (as amended into the image analysis module) is considered software and therefore does not have structure, and the validator module was not amended with any sufficient structure. Thus, the interpretation under 35 U.S.C. 112(f) will be maintained.
The rejection under 35 U.S.C. 112(b) for claims 1-12, 14, 16-17, 20-22, 24-25 have been withdrawn in light of the amended claims. However, the rejection for claim 18 will be maintained as described below.
The rejection under 35 U.S.C. 101 for claim 10 has been withdrawn in light of the Applicant’s amendments. However, the Applicant’s arguments against the rejection under 35 U.S.C. 101 for all the other claims have been considered but are unpersuasive. The Applicant states that the practical application lies in the detection of correct/incorrect product sequences and the technical feature is addressing errors of incorrect products shown in the work area. The amendments for claim 1 however, does not go above and beyond an abstract idea or a mental process (as explained below). For example, determining information of a product is something a person can do mentally, and alerting users of errors in a video is something a person (with the aid of a pencil and paper) can do mentally, such as writing down potential mistakes made in a decontamination process. The amendments containing a computer processor are also simply parts that are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea.
The Applicant’s arguments against the rejection under 35 U.S.C. 103 have been considered but are unpersuasive. The Applicant’s remarks on page 10-11 state “In the Office Action, the Examiner relies on [0019] and [0021] of Ferrante as disclosing analysing the video stream to determine when the first and second products have been presented in a correct order. However, Ferrante does not disclose analysing a video stream to identify when a first product used in a first step and a second product used in a subsequent second step are present in the work area. Further, Ferrante does not disclose using that identification to determine whether the products were presented in the correct order.”
Ferrante discloses using one or more cleaning agents and is able to detect when a subsequent cleaning agent is applied again, as shown in the procedure of Fig. 5. Although Ferrante does not disclose if the order were presented correctly, it is through Swinney’s modification (as described below) that allows Ferrante’s invention to determine such sequence.
“In paragraph [0019], Ferrante merely lists examples encompassed by the term "cleaning agent" (liquids, sprays, wipes, etc.). That is, this paragraph of Ferrante does not disclose a procedure comprising distinct first and second products used in different procedural steps, nor any ordering constraint between different products. Ferrante's listing alternatives does not teach the sequential use of multiple, different products.”.
Ferrante is shown to use one or more cleaning agents, as shown in paragraph [0019] stating “A cleaning agent may include one or more liquids, sprays, towelettes, wipes, etc., that require an operator to leave the surface sufficiently wet for a length of time and/or perform certain additional interactions with the surface”. The Applicant did not distinctly claim the first and second product to be completely different products, and only claims that there were two products involved in the video stream. Thus, due to Ferrante’s usage of multiple products, it reads on the claim limitation.
“In paragraph [0021], Ferrante describes initiation of a cleaning protocol upon detecting a wiping action. Ferrante does not disclose identifying when a first product and when a second product are present in the work area, much less tracking their temporal presentation to evaluate order. Ferrante does not disclose identifying multiple different products, associating them with distinct steps, or determining order from video evidence. Ferrante describes single-product recognition for protocol selection, not multi-product order validation. Although Ferrante may identify a type of cleaning agent (e.g., [0045]) to select or parameterize a cleaning protocol, that identification is used to assess whether a single cleaning attempt meets criteria (e.g., wetness/time). Ferrante does not disclose analysing a video stream to identify when different products appear during a procedure, nor correlating such appearances to a modelled sequence of procedural states to determine correct order. Ferrante's product recognition is not temporal sequencing of multiple products, but is instead classification to select a protocol”.
In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
“Swinney does not cure Ferrante's deficiencies. Swinney requires affirmative user action (barcode scanning) and detects order based on scan sequence, not based on analysis of a video stream identifying product presence in a work area. Swinney lacks any disclosure of video analysis, camera-based detection, or inferring presence/order from visual evidence during task performance. Substituting Swinney's barcode workflow for Ferrante's video monitoring would change the operating principle from passive, vision- based observation to active, user-driven scanning, contrary to the claimed approach”.
As additionally explained in the rejection below, the modification from Swinney is not taking the literal machine of Swinney, but rather the concept of detecting objects and determining if they are in a sequential order. While Swinney does not explicitly use a video to determine such order, Ferrante has a video that detects objects/products and the modification involves using this video to determine product sequences. Barcode readers with video cameras are well known in the art (such as the scanners used by phones to read certain barcodes in packaging companies) and thus, it would make the modification for Swinney to be fitting with Ferrante.
“The Examiner asserts it would have been obvious "to detect if Ferrante's products are correctly ordered in the video, as taught by Swinney." However, Swinney does not teach detecting order in a video, and Ferrante does not teach detecting multiple product presences or their order. The cited reference combination lacks an articulated reasoning explaining how or why a skilled artisan would (i) redesign Ferrante to perform vision-based multi-product temporal identification, (ii) discard Swinney's barcode dependency, and (iii) integrate a procedure model to determine correctness of order from video. Consequently, the cited reference combination does not discloses or suggests (1) analysing a video stream to identify when first and second products (used in different steps) are present and (2) determining (from that identification and a modelled sequence) that the products were presented in the correct order”.
As described above, the modification is not taking the machine of Swinney, but the concept of detecting if products were detected in the correct order. The identification of determining if products were presented in the correct order is a well-known procedure in day-to-day activities. They occur in things like instruction manuals or checklists, and these procedures are to ensure the most efficient methods of completing such tasks. The idea of a barcode scanner being a video camera is also well known in the art, as seen in packaging industries involving phones and scanning. Thus, it would have been obvious to combine Ferrante’s video with Swinney concept of product ordering.
Claim Objections
Claim 18 recites “A method according to claim 1.” “A method” should be read as “The method”. Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f), is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f), is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f), because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“an output device for providing audio, textual and/or visual indications to the user” in claim 20;
“an image analysis module, comprising a neural network-based classifier trained to recognize said first and second products” in claim 20;
“a validator module configured to: model the decontamination procedure as a sequence of expected transitions between procedural states” in claim 20; and
“a logging module configured to electronically log the first and second products have been presented in the correct order in the decontamination procedure” in claim 22.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f), they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f), applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. § 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1, 3-10, 12, 14, 18, 20, 22, 24-26 is/are rejected under 35 U.S.C. § 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim 1 (and correspondingly claims 20 and 24) recites “modelling the decontamination procedure as a sequence of expected transitions between procedural states”. It is unclear what modelling comprises of as the language is unclear. It is assumed the Applicant is claiming a template that determines if the decontamination procedure is going as expected. However, if this is the case, it would be expected that the model used for this template is already trained to do such templating, as it detects if the procedure is going correct by comparing it to past iterations. For examination purposes, the examiner will interpret this limitation as a model trained to identify and characterize actions that occur in the work place.
Claim 18 recites “wherein the alert event or one of the alert events comprises the first product, the second product and, when used, the third product being presented in an incorrect order in the work area”. “The third product” lacks sufficient antecedent basis. For examination purposes, the examiner will interpret this as “a third product”.
Claim 26 recites “The decontamination system according to Claim 1, wherein the modelling of the decontamination procedure as a sequence of expected transitions between procedural states is performed using a finite state machine or a Markov model”. Claim 26 is a system claim that depends on a method claim (claim 1). A method claim must depend on a method claim and vice versa with system claims. For examination purposes, the examiner will interpret claim 26 as “The method according to claim 1…”.
Claims 3-10, 12, 14, 22, 24-25 are all rejected for their dependencies on claims 1, 20, and 24 respectively.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 3 reciting “wherein analysing the video stream comprises detecting a first product container containing the first product and/or a second product container containing the second product” are rejected under 35 U.S.C. 112(d) as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. The Applicant brought up all of claim 3’s subject matter into claim 1, thus rendering claim 3 as failing to further limit the subject matter of the claim it depends on. Applicant may cancel the claims, amend the claims to place the claims in proper dependent form, rewrite the claims in independent form, or present a sufficient showing that the dependent claims complies with the statutory requirements.
Claim 6 reciting “wherein analysing the video stream further comprising determining, from the video stream, information relating to the first product and/or information relating to the second product” are rejected under 35 U.S.C. 112(d) as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. The Applicant brought up all of claim 6’s subject matter into claim 1, thus rendering claim 6 as failing to further limit the subject matter of the claim it depends on. Applicant may cancel the claims, amend the claims to place the claims in proper dependent form, rewrite the claims in independent form, or present a sufficient showing that the dependent claims complies with the statutory requirements.
Claim 7 reciting “wherein analysing the video stream comprises detecting a first product container containinq the first product and/or a second product container containinq the second product, and said determined information relating to the first product and/or said determined information relating to the second product is retrieved from a database that cross-references product information and product container appearance” are rejected under 35 U.S.C. 112(d) as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. The Applicant brought up all of claim 7’s subject matter into claim 1, thus rendering claim 7 as failing to further limit the subject matter of the claim it depends on. Applicant may cancel the claims, amend the claims to place the claims in proper dependent form, rewrite the claims in independent form, or present a sufficient showing that the dependent claims complies with the statutory requirements.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-9, 12, 14, 18, 20, 22, 24-26 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1 recites:
“capturing with the camera a video stream of a work area in which the decontamination procedure is carried out by a user” is a well-understood, routine, and conventional insignificant extra-solution activity of data gathering.
“analysing the video stream using a trained neural network executed by the computer processor to identify when the first product is present in the work area and when the second product is present in the work area” which can be reasonably interpreted as a human observer mentally analyzing if products are present in a work area. Using a computer processor and trained neural network is adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea.
“determining information relating to the first product and the second product by retrieving the information from a database that cross-references product information and product container appearance” which can be reasonably interpreted as a human observer mentally analyzing what the product container is.
“detecting using the determined information, a first product container containing the first product and a second product container containing the second product” which can be reasonably interpreted as a human observer mentally determining if a product is contained in a container.
“one or more alert events in the video stream that correspond with potential errors being made during performance of the decontamination procedure comprising an incorrect product being present in the work area” which can be reasonably interpreted as a human observer with the aid of a pencil and paper, making a list of errors in a decontamination procedure.
“modelling the decontamination procedure as a sequence of expected transitions between procedural states” which can be reasonably interpreted as a human observer with the aid of a pencil and paper, a checklist of the steps of a decontamination procedure.
“determining with the computer processor, based on said identification and said modelled sequence, that the first and second products have been presented in the a correct order in the decontamination procedure” which can be reasonably interpreted as a human observer with the aid of a pencil and paper, if products have been presented correctly in a checklist. A computer processor is adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea.
“And the computer processor causing a corresponding indication to be provided to the user, and upon identification of an alert event. the computer processor causing a corresponding alert to be provided to the user” is a well-understood, routine, and conventional insignificant extra-solution activity of data outputting.
Claim 3 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of detecting the containers of the products. A person can mentally detect the containers holding the products.
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of differentiating product containers by indicia. A person can mentally differentiate containers by their indicia.
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of differentiating product containers by their color. A person can mentally differentiate containers by their color.
Claim 6 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of determining information of products from a video. A person can mentally determine information of a product from a video, such as their material, brand, etc.
Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an additional element of gathering information retrieved from a database that has the product information and pictures. This is a well-understood, routine, and conventional insignificant extra-solution activity of data gathering.
Claim 8 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of determining information of products, whether they are approved for use in a decontamination process, and providing an indication. A person can mentally determine if certain products are approved for use, such as how strong the disinfectant is or whether the items are suitable for the process and with the aid of a pencil and paper, they can provide an indication that the product is approved.
Claim 9 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of comparing information of a product with user-input information and providing an indication to a user. A person can mentally compare information with user-input, and providing an indication to a user is a well-understood, routine, and conventional insignificant extra-solution activity of data outputting.
Claim 12 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an additional element of electronically logging information related to a product. This is a well-understood, routine, and conventional insignificant extra-solution activity of data inputting.
Claim 14 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of analyzing a video to identify a third product and determine if they are presented in the correct order of the decontamination process. A person can mentally identify a third product in a video and determine if the products are presented in the right order.
Claim 18 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of an alert comprising of when a first, second, and third products are presented in an incorrect order. A person can mentally determine if products are presented in the wrong order.
Claim 20 corresponds claim 1, additionally reciting an apparatus, camera system, output device, image analysis module, and a validator module. These parts are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Thus, claim 20 is rejected for the same reasons of being directed to an abstract idea without significantly more as claim 1.
Claim 22 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an additional element of electronically logging information related to a product’s order in the decontamination process. This is a well-understood, routine, and conventional insignificant extra-solution activity of data inputting.
Claim 24 corresponds to claim 20. Thus, they are rejected for the same reasons of being directed to an abstract idea without significantly more as claim 20.
Claim 25 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of providing the products in a container, the products being differentiated in appearance, and detecting the containers in the work area. A person can mentally differentiate the container’s appearances and detect if they are in the work area. Products being provided in a container is a well-understood, routine, and conventional activity of a product being provided in a sealed form/container (prior to being open for use).
Claim 26 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an additional element of using a Markov model/finite state machine to perform the modelling of claim 1. Using a Markov model/finite state machine is simply adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 6-8, 10, 12, 14, 18, 20, 22, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Ferrante (US 20220358761 A1) in view of Swinney (US 9433474 B2).
Regarding claim 1, Ferrante discloses a method performed by an apparatus including a computer processor and a camera (Ferrante, paragraph [0018], “A camera 105 captures a video stream comprising a continuous series of image frames and transmits the captured video stream to the cleaning wizard 150 “) for validating a decontamination procedure performed using a decontamination system (Ferrante, paragraph [0005], "The above and other issues are addressed by a cleaning wizard that monitors and provides feedback during cleaning of medical surfaces using video streaming"), the decontamination system comprising a first product for use in a first step of the procedure and a second product for use in a subsequent second step of the procedure (Ferrante, paragraph [0019], “A cleaning agent may include one or more liquids, sprays, towelettes, wipes, etc., that require an operator to leave the surface sufficiently wet for a length of time and/or perform certain additional interactions with the surface”), the method comprising:
capturing with the camera a video stream of a work area in which the decontamination procedure is carried out by a user (Ferrante, paragraph [0056], "A camera 105 captures a video stream of an item of medical equipment and transmits the video stream to a cleaning wizard 150. An operator of the item of medical equipment initiates a cleaning protocol 505A by performing a wipe of a surface of the item of medical equipment with a cloth", work area is where the camera is capturing),
analysing the video stream using a trained neural network executed by the computer processor to identify (Ferrante, paragraph [0028], "The machine learning model store 315 stores one or more models trained to output information describing meets criteria of a cleaning protocol"):
when the first product is present in the work area and when the second product is present in the work area (Ferrante, paragraph [0021], Fig. 2a below, "When an operator 215A performs a wiping motion with a cloth across a surface of the item of medical equipment 210A within the video stream captured by the camera 105A, the cleaning wizard 150 initiates a cleaning protocol for the item of medical equipment.") by:
determining information relating to the first product and the second product by retrieving the information from a database that cross-references product information and product container appearance (Ferrante, paragraph [0045], "For example, the cleaning coordination module 340 may apply a model trained to identify a type of cleaning agent and an item of medical equipment from visual data, e.g., based on a brand name or label appearing in the video stream, an application method of the cleaning agent, a spatial shape of the medical equipment, and the like."),
and detecting using the determined information, a first product container containing the first product and a second product container containing the second product (Ferrante, paragraph [0034], “Training samples may include video clips including various methods for surface cleaning, including wiping a surface with a cloth, rinsing of the surface, spraying cleaning agent on a surface, and the like”, sprays and the like naturally have a container that holds the liquid. Since the video stream captures the spraying, it will also capture the bottle holding it).
While Ferrante discloses one or more alert events in the video stream that correspond with potential errors being made during performance of the decontamination procedure (Ferrante, paragraph [0041], “The wetness prediction module 330 evaluates a pixel map corresponding to a current frame of the video stream and one or more pixel maps corresponding to one or more previous frames of the video stream, such that the wetness prediction module 330 may alert the cleaning coordination module 340 when the wetness value decreases over time to approach the threshold value”), they do not teach “comprising an incorrect product being present in the work area”.
However, Swinney teaches comprising an incorrect product being present in the work area (Swinney, paragraph [0033], "The controller will detect if the wipes are not scanned in their correct sequence or if the incorrect product is scanned. For example: [0034] The operator scans the Instrument bar code followed by the Dispenser (Activator Foam) bar code. An error bleep will sound and the display 16 will show “Incorrect product”").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to determine and display if Ferrante’s products were presented in the correct order, as taught by Swinney.
The suggestion/motivation for doing so would have been to ensure the objects/contaminated area in question is thoroughly cleaned in the most efficient sequence.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Ferrante in view of Swinney discloses modelling the decontamination procedure as a sequence of expected transitions between procedural states (Ferrante, paragraph [0028], “The machine learning model store 315 stores one or more models trained to output information describing meets criteria of a cleaning protocol”),
determining with the computer processor, based on said identification and said modelled sequence, that the first and second products have been presented in a correct order in the decontamination procedure (Swinney, Col. 5, Line 27-28, “The controller will detect if the wipes are not scanned in their correct sequence or if the incorrect product is scanned.”),
And the computer processor causing a corresponding indication to be provided to the user, and upon identification of an alert event. the computer processor causing a corresponding alert to be provided to the user (Swinney, paragraph [0034], The operator scans the Instrument bar code followed by the Dispenser (Activator Foam) bar code. An error bleep will sound and the display 16 will show “Incorrect product”").
Therefore, it would have been obvious to combine Ferrante in view of Swinney to obtain the invention as specified in claim 1.
Regarding claim 3, Ferrante in view of Swinney discloses the method according to Claim 1, wherein analysing the video stream comprises detecting a first product container containing the first product and/or a second product container containing the second product (Ferrante, paragraph [0034], “Training samples may include video clips including various methods for surface cleaning, including wiping a surface with a cloth, rinsing of the surface, spraying cleaning agent on a surface, and the like”, sprays and the like naturally have a container that holds the liquid. Since the video stream captures the spraying, it will also capture the bottle holding it).
Regarding claim 6, Ferrante in view of Swinney discloses the method according to Claim 1, wherein analysing the video stream further comprising determining, from the video stream, information relating to the first product and/or information relating to the second product (Ferrante, paragraph [0045], "For example, the cleaning coordination module 340 may apply a model trained to identify a type of cleaning agent and an item of medical equipment from visual data, e.g., based on a brand name or label appearing in the video stream, an application method of the cleaning agent, a spatial shape of the medical equipment, and the like."),
Regarding claim 7, Ferrante in view of Swinney discloses the method according to claim 6, wherein analysing the video stream comprises detecting a first product container containinq the first product and/or a second product container containinq the second product, and said determined information relating to the first product and/or said determined information relating to the second product is retrieved from a database that cross-references product information and product container appearance (Ferrante, paragraph [0045], "For example, the cleaning coordination module 340 may apply a model trained to identify a type of cleaning agent and an item of medical equipment from visual data, e.g., based on a brand name or label appearing in the video stream, an application method of the cleaning agent, a spatial shape of the medical equipment, and the like."),
Regarding claim 8, Ferrante in view of Swinney discloses the method according to Claim 6, comprising checking, using said determined information relating to the first product and/or said determined information relating to the second product, whether the first product and/or the second product are identified as approved for use in the decontamination system (Ferrante, paragraph [0045], "For example, the cleaning coordination module 340 may apply a model trained to identify a type of cleaning agent and an item of medical equipment from visual data, e.g., based on a brand name or label appearing in the video stream, an application method of the cleaning agent, a spatial shape of the medical equipment, and the like", when a model is able to identify a type of cleaning agent such as the brand name, the product will be an approved solution for a decontamination process)., and providing a corresponding indication regarding the identification to the user (Ferrante, paragraph [0051], “The first machine learning model is stored in the machine learning model store 315 and is trained to output whether the first set of frames corresponds to activity that initiates a cleaning protocol for an item of medical equipment”).
Regarding claim 10, Ferrante in view of Swinney disclose the method according to Claim 6, wherein the decontamination procedure is for a medical device (Ferrante, paragraph [0021], "In another embodiment, the camera 105A is positioned to capture a video stream including a surface on which items of medical equipment are positioned for cleaning (e.g., the camera is positioned to capture a video stream of a table on which a stethoscope is placed for cleaning purposes)"),
and wherein the method comprises checking compatibility between the medical device and the decontamination system using said determined information relating to the first product and/or said determined information relating to the second product (Ferrante, paragraph [0027], "Generally, cleaning protocols specify one or more required values or deficiency values for one or more aspects of a cleaning attempt that must be met in order for the cleaning attempt with a given cleaning agent to be successful for a given item of medical equipment, e.g., to meet CDC guidelines or manufacturer instructions for the cleaning agent").
Regarding claim 12, Ferrante in view of Swinney discloses the method according to Claim 6, comprising electronically logging said determined information relating to the first product and/or said determined information relating to the second product (Ferrante, paragraph [0026], "Alternatively, or additionally, video receipt model 305 may store the frames to shared memory accessible by the other modules of cleaning wizard 150, from which those modules may access the frames. In an embodiment, the video stream module 305 receives instructions for storing and/or transmitting the video streams from the cleaning coordination module 340.", determined information must be stored somewhere).
Regarding claim 14, Ferrante in view of Swinney discloses the method according to claim 1, wherein the decontamination system comprises a third product for use in a third step of the procedure subsequent to the second step, and wherein the method comprises: analysing the video stream to identify when the third product (Ferrante, paragraph [0019], “A cleaning agent may include one or more liquids, sprays, towelettes, wipes, etc., that require an operator to leave the surface sufficiently wet for a length of time and/or perform certain additional interactions with the surface”) is present in the work area (Ferrante, paragraph [0021], "When an operator 215A performs a wiping motion with a cloth across a surface of the item of medical equipment 210A within the video stream captured by the camera 105A, the cleaning wizard 150 initiates a cleaning protocol for the item of medical equipment.").
and determining that the first, second and third products have been presented in the correct order in the decontamination procedure (Swinney, Col. 5, Line 27-28, “The controller will detect if the wipes are not scanned in their correct sequence or if the incorrect product is scanned.”).
Regarding claim 18, Ferrante in view of Swinney discloses the method according to claim 1.
Ferrante in view of Swinney does not teach “wherein the alert event or one of the alert events comprises the first product, the second product and, when used, the third product being presented in an incorrect order in the work area”.
However, Swinney additionally teaches wherein the alert event or one of the alert events comprises the first product, the second product and, when used, the third product being presented in an incorrect order in the work area (Swinney, paragraph [0033], "The controller will detect if the wipes are not scanned in their correct sequence or if the incorrect product is scanned. For example: [0034] The operator scans the Instrument bar code followed by the Dispenser (Activator Foam) bar code. An error bleep will sound and the display 16 will show “Incorrect product”").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to display an alert on Ferrante’s (in view of Swinney) display when an incorrect product is presented in the video, as additionally taught by Swinney.
The suggestion/motivation for doing so would have been to give the user a visual that indicates something is wrong in the decontamination process and prevent them from applying the wrong product.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Ferrante in view of Swinney with the additional teachings of Swinney to obtain the invention as specified in claim 18.
Claims 20 corresponds to claim 1, additionally reciting apparatus (Ferrante, paragraph [0062], “Embodiments may also relate to an apparatus for performing the operations herein”),
a camera system (Ferrante, paragraph [0026], “The video receipt module 305 receives video streams from a camera 105”),
an output device (Ferrante, paragraph [0016], “In one embodiment, a client device 110 is a conventional computer system, such as a desktop or laptop computer. Alternatively, a client device 110 may be a device having computer functionality, such as a personal digital assistant (PDA), a mobile telephone, a smartphone, or another suitable device”),
an image analysis module (Ferrante, paragraph [0019], “The cleaning wizard 150 receives video data from the one or more cameras 105 and monitors the video stream to determine when a cleaning protocol is initiated by an operator for an item of medical equipment.”),
And a validator module (Swinney, Col. 5, Line 27-28, “The controller will detect if the wipes are not scanned in their correct sequence or if the incorrect product is scanned.”),
and to cause the output device to provide a corresponding indication to the user. (Ferrante, paragraph [0046], “The notification module 345 transmits notifications to one or more client devices 110 to provide feedback during cleaning attempts of items of medical equipment”).
Thus, claim 20 is rejected for the same reasons of obviousness as claim 1.
Regarding claim 22, Ferrante in view of Swinney discloses the apparatus according to Claim 20, further comprising a logging module configured to electronically log the first and second products have been presented in the correct order in the decontamination procedure (Ferrante, paragraph [0016], “In one embodiment, a client device 110 is a conventional computer system, such as a desktop or laptop computer”, when determining the correct sequence of products, this determination must be stored somewhere).
Claim 24 corresponds to claim 20. Thus, claim 24 is rejected for the same reasons of obviousness as claim 20 (and, correspondingly, claim 1).
Claim(s) 4-5 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Ferrante (US 20220358761 A1) in view of Swinney (US 9433474 B2) and in further view of Perron (US 20190244340 A1).
Regarding claim 4, Ferrante in view of Swinney discloses the method according to Claim 3.
Ferrante in view of Swinney does not teach “wherein the first product container and the second product container are differentiated in appearance by different indicia”.
However, Perron teaches wherein the first product container and the second product container are differentiated in appearance by different indicia (Perron, paragraph [0068], "The container database (102) may include predetermined information for other container classes as differentiated by container attributes such as size, material, color, shape, condition (e.g., undeformed, as opposed to crushed or deformed), and so forth, or mixtures of different container type", condition is a form of indicia).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to differentiate the containers of Ferrante (in view of Swinney) by their condition, as additionally taught by Perron.
The suggestion/motivation for doing so would have been to automate the classifying of the recycling process, further reducing human manual labor.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Ferrante in view of Swinney and in further view of Perron to obtain the invention as specified in claim 4.
Regarding claim 5, Ferrante in view of Swinney discloses the method according to claim 3.
Ferrante in view of Swinney does not teach “wherein the first product container and the second product container are differentiated in appearance by different colours”.
However, Perron teaches wherein the first product container and the second product container are differentiated in appearance by different colours (Perron, paragraph [0068], "The container database (102) may include predetermined information for other container classes as differentiated by container attributes such as size, material, color, shape, condition (e.g., undeformed, as opposed to crushed or deformed), and so forth, or mixtures of different container type").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to differentiate the containers of Ferrante (in view of Swinney) by their color, as taught by Perron.
The suggestion/motivation for doing so would have been to automate the classifying of the recycling process, further reducing human manual labor.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Ferrante in view of Swinney and in further view of Perron to obtain the invention as specified in claim 5.
Regarding claim 25, Ferrante in view of Swinney discloses the decontamination system according to claim 24, wherein the first product is provided in a first product container and the second product is provided in a second product container (Ferrante, paragraph [0019], “A cleaning agent may include one or more liquids, sprays, towelettes, wipes, etc., that require an operator to leave the surface sufficiently wet for a length of time and/or perform certain additional interactions with the surface”, cleaning agents such as sprays naturally come in a container).
Ferrante in view of Swinney does not teach “the first and second product containers being differentiated in appearance”.
However, Perron teaches the first and second product containers being differentiated in appearance (Perron, paragraph [0068], "The container database (102) may include predetermined information for other container classes as differentiated by container attributes such as size, material, color, shape, condition (e.g., undeformed, as opposed to crushed or deformed), and so forth, or mixtures of different container type").,
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to differentiate the container of Ferrante’s (in view of Swinney) containers by their attributes, as taught by Perron.
The suggestion/motivation for doing so would have been to automate the classification of the recycling process, further reducing human manual labor.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Ferrante in view of Swinney and Perron discloses and wherein the image analysis module is configured to detect the presence in the work area of the first product container and the second product container (Ferrante, paragraph [0034], “Training samples may include video clips including various methods for surface cleaning, including wiping a surface with a cloth, rinsing of the surface, spraying cleaning agent on a surface, and the like”, sprays and the like naturally have a container that holds the liquid. Since the video stream captures the spraying, it will also capture the bottle holding it).
Therefore, it would have been obvious to combine Ferrante in view of Swinney with Perron to obtain the invention as specified in claim 25.
Claim(s) 9 are rejected under 35 U.S.C. 103 as being unpatentable over Ferrante (US 20220358761 A1) in view of Swinney (US 9433474 B2) and in further view of Ashoori (US 20170344722 A1).
Regarding claim 9, Ferrante in view of Swinney discloses the method according to claim 6.
Ferrante in view of Swinney does not teach “comprising checking said determined information relating to the first product and/or said determined information relating to the second product against corresponding user-input information, and providing a corresponding indication to the user”.
However, Ashoori teaches comprising checking said determined information relating to the first product and/or said determined information relating to the second product against corresponding user-input information, and providing a corresponding indication to the user (Ashoori, paragraph [0005], "In an exemplary embodiment, the present invention can provide a cognitive health management method including a database, the method including analyzing user input data of a first user by comparing the user input data of the first user to medical data in the database, and providing both of: a recommendation to the first user based on the comparison of the user input data of the first user to the medical data of the database, and a result feedback including a conclusion of the analyzing to a result feedback section of the database").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to take in user input, compare it to the information of Ferrante’s (in view of Swinney) product, and provide feedback of the analysis, as taught by Ashoori.
The suggestion/motivation for doing so would have been to compare and ensure the user is utilizing the correct product during the decontamination process.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Ferrante in view of Swinney with Ashoori to obtain the invention as specified in claim 9.
Claim(s) 26 are rejected under 35 U.S.C. 103 as being unpatentable over Ferrante (US 20220358761 A1) in view of Swinney (US 9433474 B2) and in further view of Klinkigt (US 20220012502 A1).
Regarding claim 26, Ferrante in view of Swinney discloses the decontamination system according to claim 1.
Ferrante in view of Swinney does not teach “wherein the modelling of the decontamination procedure as a sequence of expected transitions between procedural states is performed using a finite state machine or a Markov model”.
However, Klinkigt teaches wherein the modelling of the decontamination procedure as a sequence of expected transitions between procedural states is performed using a finite state machine or a Markov model (Klinkigt, paragraph [0051], "More specifically, the activity detection unit 240 may generate, based on the characteristic information of the object determined by the component model unit 230, an object state corresponding to the object for each image (frame) in which the object detected in the target image sequence appears, and may detect the activity based on the transition of these object states in the Markov model.").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to use a Markov model in place of Ferrante’s (in view of Swinney) model, as taught by Klinkigt.
The suggestion/motivation for doing so would have been to have a model more suited for small datasets and not require massive amounts of data and storage for training.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Ferrante in view of Swinney and in further view of Klinkigt to obtain the invention as specified in claim 26.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WAYNE ZHANG whose telephone number is (571) 272-0245. The examiner can normally be reached Monday-Friday 10:00-6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ms. Sumati Lefkowitz can be reached on (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WAYNE ZHANG/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights