Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner’s Note
Providing supporting paragraph(s) for each limitation of amended/new claim(s) in Remarks is strongly requested for clear and definite claim interpretations by Examiner (e.g., to avoid rejections under 35 U.S.C § 112(a) “Lack of written description”)
Applicant can schedule an interview at any stage of the prosecution (e.g., Non-Final, Final, and After-Final) to discuss any issues related to, for example, rejections under 35 U.S.C § 101 and § 103, for moving toward allowance.
Priority
Acknowledgment is made of applicant's claim for the present application filed on 10/07/2021.
Response to Arguments
Applicant's arguments filed on 01/08/2026 have been fully considered but they are not persuasive.
In Remarks, pp. 9-13, Applicant contends:
1. Step 2A, Prong 1: The Claims are Not Directed to an Abstract Idea
… The specification explains that "Conceptual spaces enable the interaction between different type of data representations as an intermediate level between sub-symbolic and symbolic representations."
…
Accordingly, the claims are not directed to an abstract idea but to a specific technological solution for bridging sub-symbolic and symbolic processing in sensor processing and activity recognition systems.
…
4. Step 2A, Prong 2: The Claims Integrate the Alleged Abstract Idea into a Practical Application
… Therefore, the claims integrate any alleged abstract idea into a practical application by improving the technical field of activity recognition and timeline analysis.
…
5. Step 2B: The Claims Recite an Inventive Concept
… Thus, the claims provide significantly more than an abstract idea by reciting a non conventional improvement to activity recognition and timeline analysis systems. For the reasons above, Applicant respectfully submits that the pending claims are directed to patent-eligible subject matter.
Examiner’s response:
The examiner understands the applicant’s assertion.
However, it appears that each processing step is just applying the abstract idea to a general field of endeavor with additional elements. In addition, improvements to technology or technical field are not necessarily reflected in the claims. Thus, the claim does not integrate the judicial exception into a practical application, and the claim does not amount to significantly more than the judicial exception.
The examiner understands the applicant’s assertion “1. Step 2A, Prong 1: The Claims are Not Directed to an Abstract Idea … The specification explains that "Conceptual spaces enable the interaction between different type of data representations as an intermediate level between sub-symbolic and symbolic representations." … Accordingly, the claims are not directed to an abstract idea but to a specific technological solution for bridging sub-symbolic and symbolic processing in sensor processing and activity recognition systems.”
However, “a conceptual space” is a broad conceptual term, and “quality dimensions” could indicate any conceptual dimensions for quality measurements. In addition, a human can think of “symbolic data” and “sub-symbolic data” in his/her mind. Thus, as rejected under Claim Rejections - 35 USC § 101, other than the additional elements, the claimed limitations may be interpreted as abstract ideas.
The examiner understands the applicant’s assertion “4. Step 2A, Prong 2: The Claims Integrate the Alleged Abstract Idea into a Practical Application … Therefore, the claims integrate any alleged abstract idea into a practical application by improving the technical field of activity recognition and timeline analysis” and “a DFRE is an enhanced form of reasoning engine that further leverages the power of sub symbolic machine learning techniques, such as neural networks ( e.g., deep learning), allowing the system to operate across the full spectrum of sub symbolic data all the way to the symbolic level”.
However, “improving the technical field of activity recognition and timeline analysis” is a broad statement, and it is not clear if it is an actual improvement of the invention. In addition, it is also not clear if “enhanced form of reasoning engine that further leverages the power of sub symbolic machine learning techniques … allowing the system to operate across the full spectrum of sub symbolic data all the way to the symbolic level” is an improvement. The statement is detailed, but it sounds like a generic statement of the technologies in the related research field. Furthermore, Applicant stated “The amended claims solve these technical problems by reciting a specific multi-layer architecture that bridges sub-symbolic and symbolic processing to identify relationships between activities in order to make inferences” and “The amended claims describe a system that improves how a computer processes sensor data to recognize activities and generate meaningful timelines” (emphasis underlined) However, it appears that the statements are just about how the invention has been implemented, but they do not provide improvement clearly.
The examiner understands the applicant’s assertion “5. Step 2B: The Claims Recite an Inventive Concept … the present claims arrange sub-symbolic machine learning processing, conceptual space translation with quality dimensions, and symbolic reasoning in an unconventional multi-layer architecture that amounts to significantly more than the abstract idea itself, making it patent-eligible. Thus, the claims provide significantly more than an abstract idea by reciting a non conventional improvement to activity recognition and timeline analysis systems. For the reasons above, Applicant respectfully submits that the pending claims are directed to patent-eligible subject matter.”
However, “an unconventional multi-layer architecture” does not always make the claim patent-eligible. Even though the invention may be based on the multi-layer architecture (e.g., fig 3) towards activity recognition and timeline analysis systems, it does not appear that the claims still provide a clear improvement based on the multi-layer architecture towards activity recognition and timeline analysis systems. Providing improvements clearly based on the multi-layer architecture may help overcome the current rejections under Claim Rejections - 35 USC § 101.
For now, it is not clear why the abstract ideas are integrated into a practical application and it is not clear why the claims may include additional elements that are sufficient to amount to significantly more than the judicial exception. Rather, it appears that the limitations do not clearly show e.g., improvements in computer technology and improvements to other technical fields. It doesn’t seem that the specification and/or the independent claims clearly show how the inventive concept of the claims enables improvements and how they are tied together. The applicant may need to amend the claims to show how the claim languages and improvements are tied together, or the applicant may need to explain in more detail why the claims provide technical improvements.
To find a valid improvement to a technology, MPEP 2106.04(d)(1) says the specification must explain the improvement and that the claim must reflect the disclosed improvement. Furthermore, the improvement should not be merely a consequence of the abstract idea. See MPEP 2106.05(a). An improvement in the abstract idea itself is not an improvement to technology.
For at least these reasons, Applicant's arguments are not convincing.
Applicant’s arguments regarding 35 USC § 103 with respect to the independent claims have been considered but are moot because the arguments are directed to amended limitation(s) that has/have not been previously examined.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“A method comprising:
detecting, …, a particular activity from sensor data …;
identifying, …, relevant preceding activities to the particular activity, the identifying the relevant preceding activities including:
processing the sensor data … to generate sub-symbolic data associated with relationships in the sensor data;
translating the sub-symbolic data into symbolic data using ontologies and conceptual spaces, …; and
determining the relevant preceding activities based on the symbolic data;
generating, …, inference data indicative of a relationship of the relevant preceding activities to the particular activity; and
….”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). In particular, the claim recites an additional element(s) (“by a device”, “generated by one or more sensors in a sensor network”, “by the device and using a semantic reasoning engine implemented in a memory and executed by a processor and configured with a multi-level knowledge base”, “by the device and using the semantic reasoning engine”, “using a machine learning model”, “by the device”) – using a device and an engine to process data. The device and engine in each step are recited at a high-level of generality (i.e., as a generic computer performing a generic computer function of processing data) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“providing, by the device, an activity timeline for display that indicates the particular activity, the relevant preceding activities, and the relationship of the relevant preceding activities to the particular activity”) – the act of providing (i.e. inputting) data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of inputting data is recited at a high-level of generality (i.e., as a generic act of inputting performing a generic act function of inputting data) such that it amounts no more than a mere act to apply the exception using a generic act of inputting. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element (“the conceptual spaces each associated with quality dimensions, the symbolic data indicative of properties of a concept”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a generic computer component to perform each step amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible.
As discussed above, the claim recites the additional element(s) of inputting data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g) – “Mere Data Gathering”. However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 2
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element – “the sensor data comprises a video feed”. This is a recitation of a particular type or source of data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
This is a recitation of a particular type or source of data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 3
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). In particular, the claim recites an additional element(s) (“applying, by the device, an additional machine learning model to the sensor data”) – using a device and a model to process data. The device and model in each step are recited at a high-level of generality (i.e., as a generic computer performing a generic computer function of processing data) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a generic computer component to perform each step amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible. See MPEP 2106.05(f).
Regarding claim 4
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“… to infer whether each activity of a master activity timeline is relevant to the particular activity”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). In particular, the claim recites an additional element(s) (“using semantic reasoning engine”) – using a engine to process data. The engine in each step is recited at a high-level of generality (i.e., as a generic computer performing a generic computer function of processing data) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a generic computer component to perform each step amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible. See MPEP 2106.05(f).
Regarding claim 5
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“identifying one or more activities of the master activity timeline as irrelevant to the particular activity, wherein those one or more activities are excluded from the activity timeline provided for display”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. In particular, the claim does not recite additional elements. Thus, the claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, the claim is not patent eligible.
Regarding claim 6
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element – “the particular activity is a group activity performed by two or more people”. This is a recitation of a particular type or source of data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
This is a recitation of a particular type or source of data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 7
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“providing a safety alert based on the inference”) – the act of outputting data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of outputting data is recited at a high-level of generality (i.e., as a generic act of performing a generic act function of outputting data) such that it amounts no more than a mere act to apply the exception using a generic act of outputting. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the claim recites the additional element(s) of outputting data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
Regarding claim 8
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“inferring, …, a potential future event, based on the particular activity and at least one of the relevant preceding activities”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). In particular, the claim recites an additional element(s) (“using semantic reasoning engine”) – using an engine to process data. The engine in each step is recited at a high-level of generality (i.e., as a generic computer performing a generic computer function of processing data) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a generic computer component to perform each step amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible. See MPEP 2106.05(f).
Regarding claim 9
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). In particular, the claim recites an additional element(s) (“by the device”) – using a device to process data. The device in each step is recited at a high-level of generality (i.e., as a generic computer performing a generic computer function of processing data) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“providing, by the device, at least a portion of the sensor data for display in conjunction with the activity timeline”) – the act of providing data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of sending data is recited at a high-level of generality (i.e., as a generic act of sending performing a generic act function of sending data) such that it amounts no more than a mere act to apply the exception using a generic act of sending. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a generic computer component to perform each step amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible.
As discussed above, the claim recites the additional element (“providing, by the device, at least a portion of the sensor data for display in conjunction with the activity timeline”) at a high-level of generality and is adding an insignificant extra-solution activity (i.e. pre-solution activity) – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
Regarding claim 10
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“identifying a person associated with the particular activity across sensor data …”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element (“from a plurality of sensors”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 11
The claim recites “An apparatus, comprising: a network interface circuit configured to receive sensor data from a computer network; a processor coupled to the network interface circuit; and a memory configured to store one or more instructions, that when executed by the processor, configure the processor to:” to perform precisely the method of Claim 1. As performance of an abstract idea on generic computer components (see MPEP 2106.05(f)) and “Storing and retrieving information in memory” (see MPEP 2106.05(g) on Insignificant Extra-Solution Activity, and MPEP 2106.05(d) on Well-Understood, Routine, Conventional Activity) and “Receiving or transmitting data over a network” (see MPEP 2106.05(g) on Insignificant Extra-Solution Activity, and MPEP 2106.05(d) on Well-Understood, Routine, Conventional Activity) cannot integrate the abstract idea into a practical application nor provide significantly more than the abstract idea itself, the claim is rejected for reasons set forth in the rejection of Claim 1.
Regarding claim 12
The claim is rejected for the reasons set forth in the rejection of Claim 2 under 35 U.S.C. 101, mutatis mutandis, as reciting an abstract idea without integrating the judicial exception into a practical application nor providing significantly more than the judicial exception.
Regarding claim 13
The claim is rejected for the reasons set forth in the rejection of Claim 3 under 35 U.S.C. 101, mutatis mutandis, as reciting an abstract idea without integrating the judicial exception into a practical application nor providing significantly more than the judicial exception.
Regarding claim 14
The claim is rejected for the reasons set forth in the rejection of Claim 4 under 35 U.S.C. 101, mutatis mutandis, as reciting an abstract idea without integrating the judicial exception into a practical application nor providing significantly more than the judicial exception.
Regarding claim 15
The claim is rejected for the reasons set forth in the rejection of Claim 5 under 35 U.S.C. 101, mutatis mutandis, as reciting an abstract idea without integrating the judicial exception into a practical application nor providing significantly more than the judicial exception.
Regarding claim 16
The claim is rejected for the reasons set forth in the rejection of Claim 6 under 35 U.S.C. 101, mutatis mutandis, as reciting an abstract idea without integrating the judicial exception into a practical application nor providing significantly more than the judicial exception.
Regarding claim 17
The claim is rejected for the reasons set forth in the rejection of Claim 7 under 35 U.S.C. 101, mutatis mutandis, as reciting an abstract idea without integrating the judicial exception into a practical application nor providing significantly more than the judicial exception.
Regarding claim 18
The claim is rejected for the reasons set forth in the rejection of Claim 8 under 35 U.S.C. 101, mutatis mutandis, as reciting an abstract idea without integrating the judicial exception into a practical application nor providing significantly more than the judicial exception.
Regarding claim 19
The claim is rejected for the reasons set forth in the rejection of Claim 9 under 35 U.S.C. 101, mutatis mutandis, as reciting an abstract idea without integrating the judicial exception into a practical application nor providing significantly more than the judicial exception.
Regarding claim 20
The claim recites “A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising” to perform precisely the method of Claim 1. As performance of an abstract idea on generic computer components (see MPEP 2106.05(f)) and “Storing and retrieving information in memory” (see MPEP 2106.05(g) on Insignificant Extra-Solution Activity, and MPEP 2106.05(d) on Well-Understood, Routine, Conventional Activity) cannot integrate the abstract idea into a practical application nor provide significantly more than the abstract idea itself, the claim is rejected for reasons set forth in the rejection of Claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 7, 9, 11-15, 17, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (Temporal image analytics for abnormal construction activity identification) in view of Kerzner et al. (US 11,195,311 B1)
Regarding claim 1
(Note: Hereinafter, if a limitation has bold brackets (i.e. [·]) around claim languages, the bracketed claim languages indicate that they have not been taught yet by the current prior art reference but they will be taught by another prior art reference afterwards.)
Lin teaches
A method comprising:
detecting, by a device, a particular activity from sensor data generated by one or more sensors in a sensor [network];
(Lin [fig(s) 1-2] [sec(s) 3] “This study follows the typical pipeline consisting the following four steps: object detection, object tracking, action recognition, and operational analysis (Fig. 1). The objects of interest, such as workers, excavators and dump trucks, are detected in images. Second, detection results are associated in attempt to distinguish individual trajectories. In other words, tracking is conducted. Third, the time-series data obtained from the previous module are used to recognize particular actions.” [sec(s) 5.1] “in this research, the data are in the format of sequential images from surveillance cameras.” [sec(s) 4.3] “In the dataset [5], activities of excavators and dump trucks were manually annotated according to timeline. As excavators are one of the primary participants in the earthmoving operation, each action performed by excavators corresponds to a coded number: 0 = Idling, 1 = Moving, 2 = Swinging, 3 = Digging, and 4 = Dumping. The activities of excavators and dump trucks in the videos were documented into the line chart.” [sec(s) 4.1] “The training was conducted on a computer with a Linux Ubuntu 18.04 operating system, AMD Ryzen™ 72,700× CPU, GeForce RTX2080Ti GPU, and 32G RAM.”;)
identifying, by the device and using a semantic reasoning engine implemented in a memory and executed by a processor and configured with a multi-level knowledge base, relevant preceding activities to the particular activity, the identifying the relevant preceding activities including:
(Lin [fig(s) 1] [fig(s) 2] “Alarm” [fig(s) 13] [sec(s) 5.1] “They are filtered and marked in yellow, and the remaining cycles marked in red can be manually inspected by the managers. For identification of irregular events, the action sequence and cycle time are two major factors utilized in this work.” [sec(s) 3.4] “If the cycle time of an operation is greater than a threshold and the action sequence differs from operation samples, the operation will be considered irregular. … The training was conducted on a computer with a Linux Ubuntu 18.04 operating system, AMD Ryzen™ 72,700× CPU, GeForce RTX2080Ti GPU, and 32G RAM.” [sec(s) 4.1] “The first task is to recognize heavy equipment, namely excavators and dump trucks, during construction. To augment the detection performance for the Faster R-CNN, the Alberta Construction Image Dataset (ACID) [66] was adopted for transfer learning of the Faster R-CNN model. The ACID contains 2850 images, including 6058 instances, among which 2388 are excavators, 2835 are dump trucks, and 835 are concrete mixer trucks. Some images from ACID are shown in Fig. 4.” [sec(s) 4.4.1] “Fig. 13 shows two line charts of the two testing videos with irregular operations marked in red or yellow. The red lines represent irregular operations which may be abnormal and should be examined by the field manager. We call these irregular operations potential abnormal activities.” [sec(s) 6] “Action recognition takes ground truth annotations from the previous work, and the line chart is produced. In the line chart, irregular operation cycles were identified and marked in red. The irregular events are in accordance with action sequence and cycle time. The action sequences follow the definition in previous studies, while the statistics and the box plot facilitate identification of irregular cycle time.” [fig(s) C.21] [sec(s) Appendix C] “The bounding box annotations are speculated to be a possible reason contributing to the low AP. If part of the dump truck had not been in the photo, only the rest was labeled. As shown in Fig. C.21, some dump trucks had only the front part labeled, yet the other had only the dump box labeled. Therefore, the detector might not be capable enough to exquisitely predict the position of the bounding boxes. To increase the AP, filtering photos with incomplete targets may be a solution” [fig(s) D.22] [sec(s) Appendix D] “Taking video No.104154 as an example, the ground truth annotation contains only the working equipment. … Those objects not contained in the ground truth annotation might decrease the MOTA”; e.g., fig 2 read(s) on “semantic reasoning engine”. In addition, e.g., “For identification of irregular events, the action sequence and cycle time are two major factors” read(s) on “identifying … relevant preceding activities” since “the action sequence” is used for deciding an irregular operation. Furthermore, e.g., “Alberta Construction Image Dataset (ACID)” along with images and labeled ground truth annotations (e.g., fig C.21) read(s) on “multi-level knowledge base”, and e.g., “images” read(s) on “sub-symbolic sensor features” and e.g., representations (e.g., annotations) for “excavators … dump trucks … concrete mixer trucks” read(s) on “symbolic data representations” and “conceptual”.
Examiner notes that page 11 of the Instant Specification describes “At the lowest layer of hierarchy 300 is sub-symbolic layer 302 that processes the sensor data 312 collected from the network. For example, sensor data 312 may include video feed/stream data from any number of cameras located throughout a location. In some embodiments, sensor data 312 may comprise multimodal sensor data from any number of different types of sensors located throughout the location. At the core of sub-symbolic layer 302 may be one or more DNNs 308 or other machine learning-based model that processes the collected sensor data 312. In other words, sub-symbolic layer 302 may perform sensor fusion on sensor data 312 to identify hidden relationships between the data.”
Examiner notes that page 14 of the Instant Specification describes “In other words, a DFRE generally refers to a cognitive engine capable of taking sub-symbolic data as input (e.g., raw or processed sensor data regarding a monitored system), recognizing symbolic concepts from that data, and applying symbolic reasoning to the concepts, to draw conclusions about the monitored system.”)
processing the sensor data using a machine learning model to generate sub-symbolic data associated with relationships in the sensor data;
(Lin [fig(s) 1] [fig(s) 2] [fig(s) 13] [sec(s) 5.1] “They are filtered and marked in yellow, and the remaining cycles marked in red can be manually inspected by the managers. For identification of irregular events, the action sequence and cycle time are two major factors utilized in this work.” [sec(s) 3.4] “If the cycle time of an operation is greater than a threshold and the action sequence differs from operation samples, the operation will be considered irregular.” [sec(s) 4.1] “The first task is to recognize heavy equipment, namely excavators and dump trucks, during construction. To augment the detection performance for the Faster R-CNN, the Alberta Construction Image Dataset (ACID) [66] was adopted for transfer learning of the Faster R-CNN model. The ACID contains 2850 images, including 6058 instances, among which 2388 are excavators, 2835 are dump trucks, and 835 are concrete mixer trucks. Some images from ACID are shown in Fig. 4.” [sec(s) 4.4.1] “Fig. 13 shows two line charts of the two testing videos with irregular operations marked in red or yellow. The red lines represent irregular operations which may be abnormal and should be examined by the field manager. We call these irregular operations potential abnormal activities.” [sec(s) 6] “Action recognition takes ground truth annotations from the previous work, and the line chart is produced. In the line chart, irregular operation cycles were identified and marked in red. The irregular events are in accordance with action sequence and cycle time. The action sequences follow the definition in previous studies, while the statistics and the box plot facilitate identification of irregular cycle time.” [fig(s) C.21] [sec(s) Appendix C] “The bounding box annotations are speculated to be a possible reason contributing to the low AP. If part of the dump truck had not been in the photo, only the rest was labeled. As shown in Fig. C.21, some dump trucks had only the front part labeled, yet the other had only the dump box labeled. Therefore, the detector might not be capable enough to exquisitely predict the position of the bounding boxes. To increase the AP, filtering photos with incomplete targets may be a solution”; e.g., “images” read(s) on “sub-symbolic data”.)
translating the sub-symbolic data into symbolic data using ontologies and conceptual spaces, the conceptual spaces each associated with quality dimensions, the symbolic data indicative of properties of a concept; and
(Lin [fig(s) 1] [fig(s) 2] [fig(s) 13] [sec(s) 5.1] “They are filtered and marked in yellow, and the remaining cycles marked in red can be manually inspected by the managers. For identification of irregular events, the action sequence and cycle time are two major factors utilized in this work.” [sec(s) 3.4] “If the cycle time of an operation is greater than a threshold and the action sequence differs from operation samples, the operation will be considered irregular.” [sec(s) 4.1] “The first task is to recognize heavy equipment, namely excavators and dump trucks, during construction. To augment the detection performance for the Faster R-CNN, the Alberta Construction Image Dataset (ACID) [66] was adopted for transfer learning of the Faster R-CNN model. The ACID contains 2850 images, including 6058 instances, among which 2388 are excavators, 2835 are dump trucks, and 835 are concrete mixer trucks. Some images from ACID are shown in Fig. 4.” [sec(s) 4.4.1] “Fig. 13 shows two line charts of the two testing videos with irregular operations marked in red or yellow. The red lines represent irregular operations which may be abnormal and should be examined by the field manager. We call these irregular operations potential abnormal activities.” [sec(s) 6] “Action recognition takes ground truth annotations from the previous work, and the line chart is produced. In the line chart, irregular operation cycles were identified and marked in red. The irregular events are in accordance with action sequence and cycle time. The action sequences follow the definition in previous studies, while the statistics and the box plot facilitate identification of irregular cycle time.” [fig(s) C.21] [sec(s) Appendix C] “The bounding box annotations are speculated to be a possible reason contributing to the low AP. If part of the dump truck had not been in the photo, only the rest was labeled. As shown in Fig. C.21, some dump trucks had only the front part labeled, yet the other had only the dump box labeled. Therefore, the detector might not be capable enough to exquisitely predict the position of the bounding boxes. To increase the AP, filtering photos with incomplete targets may be a solution” [sec(s) 4.4.3] “In other words, the process of detection, tracking, and action recognition is performed in real-time, and the new operation cycle will be compared with the statistical data to determine if the new operation cycle is irregular.”; e.g., “images” read(s) on “sub-symbolic data”. In addition, e.g., representations (e.g., annotations) for “excavators … dump trucks … concrete mixer trucks” read(s) on “symbolic data”. Furthermore, e.g., performance read(s) on “quality”. Moreover, e.g., a set of concepts and categories for construction activities read(s) on “ontologies”.)
determining the relevant preceding activities based on the symbolic data;
(Lin [fig(s) 1] [fig(s) 2] [fig(s) 13] [sec(s) 5.1] “They are filtered and marked in yellow, and the remaining cycles marked in red can be manually inspected by the managers. For identification of irregular events, the action sequence and cycle time are two major factors utilized in this work.” [sec(s) 3.4] “If the cycle time of an operation is greater than a threshold and the action sequence differs from operation samples, the operation will be considered irregular.” [sec(s) 4.1] “The first task is to recognize heavy equipment, namely excavators and dump trucks, during construction. To augment the detection performance for the Faster R-CNN, the Alberta Construction Image Dataset (ACID) [66] was adopted for transfer learning of the Faster R-CNN model. The ACID contains 2850 images, including 6058 instances, among which 2388 are excavators, 2835 are dump trucks, and 835 are concrete mixer trucks. Some images from ACID are shown in Fig. 4.” [sec(s) 4.4.1] “Fig. 13 shows two line charts of the two testing videos with irregular operations marked in red or yellow. The red lines represent irregular operations which may be abnormal and should be examined by the field manager. We call these irregular operations potential abnormal activities.” [sec(s) 6] “Action recognition takes ground truth annotations from the previous work, and the line chart is produced. In the line chart, irregular operation cycles were identified and marked in red. The irregular events are in accordance with action sequence and cycle time. The action sequences follow the definition in previous studies, while the statistics and the box plot facilitate identification of irregular cycle time.” [fig(s) C.21] [sec(s) Appendix C] “The bounding box annotations are speculated to be a possible reason contributing to the low AP. If part of the dump truck had not been in the photo, only the rest was labeled. As shown in Fig. C.21, some dump trucks had only the front part labeled, yet the other had only the dump box labeled. Therefore, the detector might not be capable enough to exquisitely predict the position of the bounding boxes. To increase the AP, filtering photos with incomplete targets may be a solution” [sec(s) 4.4.3] “In other words, the process of detection, tracking, and action recognition is performed in real-time, and the new operation cycle will be compared with the statistical data to determine if the new operation cycle is irregular.”; e.g., representations (e.g., annotations) for “excavators … dump trucks … concrete mixer trucks” read(s) on “symbolic data”. Furthermore, e.g., “For identification of irregular events, the action sequence and cycle time are two major factors” read(s) on “determining relevant preceding activities” since “the action sequence” is used for deciding an irregular operation.)
generating, by the device and using the semantic reasoning engine, inference data indicative of a relationship of the relevant preceding activities to the particular activity; and
(Lin [fig(s) 1] [fig(s) 2] “Alarm” [fig(s) 13] [sec(s) 5.1] “For example, irregular operations such as truck exchanges are not abnormal. They are filtered and marked in yellow, and the remaining cycles marked in red can be manually inspected by the managers. For identification of irregular events, the action sequence and cycle time are two major factors utilized in this work. A different action sequence of an operation cycle does not indicate abnormal activity” [sec(s) 4.4.1] “Fig. 13 shows two line charts of the two testing videos with irregular operations marked in red or yellow. The red lines represent irregular operations which may be abnormal and should be examined by the field manager. We call these irregular operations potential abnormal activities. The yellow lines indicate irregular operations that may not be critical yet can be leveraged for productivity improvement, such as better truck exchange.” [fig(s) C.21] [sec(s) Appendix C] “The bounding box annotations are speculated to be a possible reason contributing to the low AP. If part of the dump truck had not been in the photo, only the rest was labeled. As shown in Fig. C.21, some dump trucks had only the front part labeled, yet the other had only the dump box labeled. Therefore, the detector might not be capable enough to exquisitely predict the position of the bounding boxes. To increase the AP, filtering photos with incomplete targets may be a solution” [fig(s) D.22] [sec(s) Appendix D] “Taking video No.104154 as an example, the ground truth annotation contains only the working equipment. … Those objects not contained in the ground truth annotation might decrease the MOTA”; e.g., fig 2 read(s) on “semantic reasoning engine”. In addition, e.g., “For identification of irregular events, the action sequence and cycle time are two major factors” read(s) on “relationship of the relevant preceding activities to the particular activity” since “the action sequence” is used for deciding an irregular operation. Furthermore, e.g., “Alberta Construction Image Dataset (ACID)” along with images having construction site objects and labeled ground truth annotations for them (e.g., fig C.21) read(s) on “correlating data at different abstraction levels in the multi-level knowledge base”.)
providing, by the device, an activity timeline for display that indicates the particular activity, the relevant preceding activities, and the relationship of the relevant preceding activities to the particular activity.
(Lin [fig(s) 1] [fig(s) 2] “Alarm” [fig(s) 13] [sec(s) 5.1] “For example, irregular operations such as truck exchanges are not abnormal. They are filtered and marked in yellow, and the remaining cycles marked in red can be manually inspected by the managers. For identification of irregular events, the action sequence and cycle time are two major factors utilized in this work. A different action sequence of an operation cycle does not indicate abnormal activity” [sec(s) 4.4.1] “Fig. 13 shows two line charts of the two testing videos with irregular operations marked in red or yellow. The red lines represent irregular operations which may be abnormal and should be examined by the field manager. We call these irregular operations potential abnormal activities. The yellow lines indicate irregular operations that may not be critical yet can be leveraged for productivity improvement, such as better truck exchange.” [sec(s) 3.4] “In practice, the industry utilizes the CBC to study construction activities. For a more effective visualization, the construction process and irregular operations could be depicted through the line chart as an alternative form of the CBC. As demonstrated in Fig. 3, the x-axis represents the timeline, and the y-axis indicates actions. The actions of each worker or equipment are documented according to the timeline with different colors. For a timely identification, classified irregular events are pre-screened and highlighted with distinctive colors.”; e.g., fig 2 read(s) on “semantic reasoning engine”. In addition, e.g., “For identification of irregular events, the action sequence and cycle time are two major factors” along with figs 1, 13, 17-18, F.25 read(s) on “relationship of the relevant preceding activities to the particular activity” since “the action sequence” is used for deciding an irregular operation.)
However, Lin does not appear to explicitly teach:
detecting, by a device, a particular activity from sensor data generated by one or more sensors in a sensor [network];
(Note: Hereinafter, if a limitation has one or more bold underlines, the one or more underlined claim languages indicate that they are taught by the current prior art reference, while the one or more non-underlined claim languages indicate that they have been taught already by one or more previous art references.)
Kerzner teaches
detecting, by a device, a particular activity from sensor data generated by one or more sensors in a sensor network;
(Kerzner [fig(s) 1A, 1B] [col 2 ln 1– col 6 ln 9] “The timeline panel 12 includes a timeline that represents density of events detected by the monitoring system. The density of events may refer to the number of events detected within a particular range of time. … For example, an event indicating a door leading to outside the house was opened may be more important than an event indicating a door between rooms of the house was opened. Accordingly, the event indicating a door leading to outside the house was opened may be associated with a weight that is three times greater, e.g., 0.6, than the weight, e.g., 0.2, associated with the event indicating a door between rooms of the house was opened. … The stream panel 16 may also be updated to show camera item 20 that shows that the latest event before the event item 18 was storing new video clips from an office camera and show camera item 22 that shows that the second latest event before the event item 18 was capturing images of the front door.” [col 8 ln 58– col 9 ln 8] “The sensors 120 may include a contact sensor, a motion sensor, a glass break sensor, or any other type of sensor included in an alarm system or security system. The sensors 120 also may include an environmental 65 sensor, such as a temperature sensor, a water sensor, a rain sensor, a wind sensor, a light sensor, a smoke detector, a carbon monoxide detector, an air quality sensor, etc.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Lin with the sensor network of Kerzner.
One of ordinary skill in the art would have been motived to combine in order to provide increased security and intelligently present events detected by a monitoring system so that a user may more easily understand events detected by the monitoring system.
(Kerzner [col 1 ln 23– col 2 ln 26] “Many people equip homes and businesses with alarm systems to provide increased security for their homes and businesses. … the analysis engine may intelligently present events detected by the monitoring system so that a user may more easily understand events detected by the monitoring system.”)
Regarding claim 2
The combination of Lin, Kerzner teaches claim 1.
Lin further teaches
the sensor data comprises a video feed.
(Lin [fig(s) 1-2] “Video” [sec(s) 5.1] “in this research, the data are in the format of sequential images from surveillance cameras.” [sec(s) 3.2] “In this section, detection results from the last module are then associated to form sequential information of each object in the video. The individual object will have its ID and consecutive locations in the image sequence.”;)
Regarding claim 3
The combination of Lin, Kerzner teaches claim 1.
Lin further teaches
detecting the particular activity comprises:
applying, by the device, an additional machine learning model to the sensor data.
(Lin [fig(s) 1-2] [sec(s) 3] “This study follows the typical pipeline consisting the following four steps: object detection, object tracking, action recognition, and operational analysis (Fig. 1). The objects of interest, such as workers, excavators and dump trucks, are detected in images. Second, detection results are associated in attempt to distinguish individual trajectories. In other words, tracking is conducted. Third, the time-series data obtained from the previous module are used to recognize particular actions.” [sec(s) 3.1-3.4] “Since the Faster R-CNN relies on features computed only from a single scale [42], the integration of FPN can increase the performance on detecting objects at different scales without intensive computation [46]. … Since SORT considers only the motion features, it was 20× faster than other state-of-the-art trackers, such as Markov Decision Processes (MDPs) [52] and temporal dynamic appearance model (TDAM) [49,53]. … To address the issue of re-identification, the authors modified SORT for better object tracking performance. … The action recognition module utilizes a CNN that takes tracking trajectories and extracts the visual features, and utilizes a LSTM for learning of the sequential patterns [3,26]. This module outputs the recognized actions.” [sec(s) 4.1] “The training was conducted on a computer with a Linux Ubuntu 18.04 operating system, AMD Ryzen™ 72,700× CPU, GeForce RTX2080Ti GPU, and 32G RAM.”;)
Regarding claim 4
The combination of Lin, Kerzner teaches claim 1.
Lin further teaches
identifying the relevant preceding activities comprises: (see the rejections of claim 1)
Kerzner teaches
using the semantic reasoning engine to infer whether each activity of a master activity timeline is relevant to the particular activity.
(Kerzner [fig(s) 1A, 1B] “TIMELINE” [col 5 ln 31– col 7 ln 41] “The selection element for types of events may enable a user to toggle particular types of events to display in the timeline and for display of monitoring system data in the stream panel 16. For instance, if the user selects to toggle off events related to lights, the analysis engine may display a timeline that ignores events with the event type of lights. The types of events may include security system, locks, lights, video, thermostats, and others.” and “The analysis engine can group a series of events associated with an event type for display. For example, if a user has selected to filter by Waking Up events for a period of one month, a single image, video, or other monitoring data representative of Waking Up for each day can be shown. In some implementations, a single image, video, or portion of monitoring data can be shown for all events of an event type.” [col 13 ln 55– col 14 ln 67] “the monitoring system control unit 110 performs the processing of the analysis engine.” [col 21 ln 4-col21 ln 64] “For example, the user can select several Lights On events to create a new event type Waking Up. The system 500 can learn from the user's interactions with the generated graphical representation. In some implementations, the system 500 curates events and adjusts weights based on the user's interactions with the generated graphical representation.”; e.g., a collection of all event data read(s) on “master activity timeline”. In addition, e.g., “enable a user to toggle particular types of events to display in the timeline” read(s) on “infer whether each activity of a master activity timeline is relevant to the particular activity” since only relevant events are displayed based on inference.)
The combination of Lin, Kerzner is combinable with Kerzner for the same rationale as set forth above with respect to claim 1.
Regarding claim 5
The combination of Lin, Kerzner teaches claim 4.
Kerzner further teaches
using the semantic reasoning engine to infer whether each activity of the master activity timeline is relevant to the particular activity comprises: (see the rejections of claim 4)
Kerzner further teaches
identifying one or more activities of the master activity timeline as irrelevant to the particular activity, wherein the one or more activities are excluded from the activity timeline provided for display.
(Kerzner [fig(s) 1A, 1B] “TIMELINE” [col 5 ln 31– col 7 ln 41] “The selection element for types of events may enable a user to toggle particular types of events to display in the timeline and for display of monitoring system data in the stream panel 16. For instance, if the user selects to toggle off events related to lights, the analysis engine may display a timeline that ignores events with the event type of lights. The types of events may include security system, locks, lights, video, thermostats, and others.” and “The analysis engine can group a series of events associated with an event type for display. For example, if a user has selected to filter by Waking Up events for a period of one month, a single image, video, or other monitoring data representative of Waking Up for each day can be shown. In some implementations, a single image, video, or portion of monitoring data can be shown for all events of an event type.” [col 13 ln 55– col 14 ln 67] “the monitoring system control unit 110 performs the processing of the analysis engine.”; e.g., an activity that the user has selected to display (e.g., “locks”) read(s) on “particular activity”. In addition, e.g., “events related to lights” read(s) on “one or more activities of the master activity timeline as irrelevant to the particular activity” since an activity that the user has selected to display (e.g., “locks”) is not relevant to “events related to lights”.)
The combination of Lin, Kerzner is combinable with Kerzner for the same rationale as set forth above with respect to claim 1.
Regarding claim 7
The combination of Lin, Kerzner teaches claim 1.
Lin further teaches
providing a safety alert based on the inference data.
(Lin [fig(s) 1] [fig(s) 2] “Alarm” [sec(s) Abs] “Abnormal activities on construction jobsites may compromise productivity and pose threat to workers’ safety. This paper proposes the analysis of consecutive image sequences for automatic identification of irregular operations and their visualization.” [sec(s) 1] “Irregular operations are potential abnormal events that may decrease productivity and may be related to construction safety. With the provided information, field manager can directly acquire surveillance videos of irregular operations to investigate the cause of anomalies.” [fig(s) 13] [sec(s) 5.1] “For example, irregular operations such as truck exchanges are not abnormal. They are filtered and marked in yellow, and the remaining cycles marked in red can be manually inspected by the managers. For identification of irregular events, the action sequence and cycle time are two major factors utilized in this work.” [sec(s) 5.2] “Based on the best knowledge of the authors, little research has attempted to identify irregular operations, which are intimately related to progress and safety and should be thoroughly inspected.” [sec(s) 4.4.1] “Fig. 13 shows two line charts of the two testing videos with irregular operations marked in red or yellow. The red lines represent irregular operations which may be abnormal and should be examined by the field manager. We call these irregular operations potential abnormal activities. The yellow lines indicate irregular operations that may not be critical yet can be leveraged for productivity improvement, such as better truck exchange.”; e.g., “Alarm” read(s) on “safety alert”.)
Regarding claim 9
The combination of Lin, Kerzner teaches claim 1.
Kerzner further teaches
providing, by the device, at least a portion of the sensor data for display in conjunction with the activity timeline.
(Kerzner [fig(s) 1A, 1B] [col 2 ln 1– col 6 ln 9] “The timeline panel 12 includes a timeline that represents density of events detected by the monitoring system. The density of events may refer to the number of events detected within a particular range of time. … For example, an event indicating a door leading to outside the house was opened may be more important than an event indicating a door between rooms of the house was opened. Accordingly, the event indicating a door leading to outside the house was opened may be associated with a weight that is three times greater, e.g., 0.6, than the weight, e.g., 0.2, associated with the event indicating a door between rooms of the house was opened. … The stream panel 16 may also be updated to show camera item 20 that shows that the latest event before the event item 18 was storing new video clips from an office camera and show camera item 22 that shows that the second latest event before the event item 18 was capturing images of the front door.”;)
The combination of Lin, Kerzner is combinable with Kerzner for the same rationale as set forth above with respect to claim 1.
Regarding claim 11
The claim is a system claim corresponding to the method claim 1, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Note that Kerzner teaches
An apparatus, comprising:
a network interface circuit configured to receive sensor data from a computer network;
(Kerzner [fig(s) 2] [col 7 ln 41– col 10 ln 61] “The network module 114 is a communication device configured to exchange communications over the network 105. … The network module 114 also may be a wired communication module configured to exchange communications over the network 105 using a wired connection. For instance, the network module 114 may be a modem, a network interface card, or another type of network interface device.”)
a processor coupled to the network interface circuit; and
(Kerzner [fig(s) 2] [col 7 ln 41– col 10 ln 61] “In some examples, the controller 112 may include a processor or other control circuitry configured to execute instructions of a program that controls operation of an alarm system. … The network module 114 is a communication device configured to exchange communications over the network 105. … The network module 114 also may be a wired communication module configured to exchange communications over the network 105 using a wired connection. For instance, the network module 114 may be a modem, a network interface card, or another type of network interface device.” [col 14 ln 4– col 14 ln 52] “For example, rather than being a separate server located in a remote location, the monitoring application server 160 may be a logical component inside of the monitoring system control unit 110.”)
a memory configured to store one or more instructions, that when executed by the processor, configure the processor to:
(Kerzner [fig(s) 2] [col 23 ln 6– col 23 ln 47] “Generally, a processor will receive instructions and data from a read-only memory and/or a random access memory. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and Compact Disc Read-Only Memory (CD-ROM). Any of the foregoing may be supplemented by, or incorporated in, specially-designed ASICs (application-specific integrated circuits).”)
The combination of Lin, Kerzner is combinable with Kerzner for the same rationale as set forth above with respect to claim 1.
Regarding claim 12
The claim is a system claim corresponding to the method claim 2, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 13
The claim is a system claim corresponding to the method claim 3, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 14
The claim is a system claim corresponding to the method claim 4, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 15
The claim is a system claim corresponding to the method claim 5, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 17
The claim is a system claim corresponding to the method claim 7, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 19
The claim is a system claim corresponding to the method claim 9, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 20
The claim is a computer-readable medium claim corresponding to the method claim 1, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Note that Kerzner teaches “computer-readable medium”.
(Lin [sec(s) 4.1] “The training was conducted on a computer with a Linux Ubuntu 18.04 operating system, AMD Ryzen™ 72,700× CPU, GeForce RTX2080Ti GPU, and 32G RAM.”;)
Claim(s) 6, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (Temporal image analytics for abnormal construction activity identification) in view of Kerzner et al. (US 11,195,311 B1) in view of Pei et al. (Learning and parsing video events with goal and intent prediction)
Regarding claim 6
The combination of Lin, Kerzner teaches claim 1.
However, the combination of does not appear to explicitly teach:
the particular activity is a group activity performed by two or more people.
Pei teaches
the particular activity is a group activity performed by two or more people.
(Pei [table(s) 2] “Discussion” [fig(s) 15] “Experiment results of event parsing for multiple agents. Agent P1 works during frames 4861 to 6196, agent P2 enters the room from frames 6000 to 6196, then they go to the white board, have a discussion and leave the board.” [sec(s) 2.2] “Table 2 shows the atomic actions used in the office scene. These atomic actions are learned automatically from the training data. The learning process is explained in Section 3.” [sec(s) 5.1] “The testing video lasts 50 min and contains 12 event categories, including single-agent events like getting water and using a microwave, and multi-agent events like discussing at the white board and exchanging objects. The testing video also includes event insertion such as making a call while getting water.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Lin, Kerzner with the group activity by people of Pei.
One of ordinary skill in the art would have been motived to combine in order to improve the detection result of atomic actions, and to better segment and recognize objects in the scene.
(Pei [sec(s) Abs] “The algorithm uses event context to improve the detection of atomic actions, segment and recognize objects in the scene. Extensive experiments, including indoor and out door scenes, single and multiple agents events, are conducted to validate the effectiveness of the proposed approach.” [sec(s) 1] “We show that event context can be used to improve the detection result of atomic actions, and to better segment and recognize objects in the scene.” [sec(s) 5] “From the ROC curve we can see that with event context, the recognition rate of atomic actions is improved greatly.”)
Regarding claim 16
The claim is a system claim corresponding to the method claim 6, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Claim(s) 8, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (Temporal image analytics for abnormal construction activity identification) in view of Kerzner et al. (US 11,195,311 B1) in view of Koppula et al. (Anticipating Human Activities Using Object Affordances for Reactive Robotic Response)
Regarding claim 8
The combination of Lin, Kerzner teaches claim 1.
However, the combination of Lin, Kerzner does not appear to explicitly teach:
inferring, using the semantic reasoning engine, a potential future event, based on the particular activity and at least one of the relevant preceding activities.
Koppula teaches
inferring, using the semantic reasoning engine, a potential future event, based on the particular activity and at least one of the relevant preceding activities.
(Koppula [fig(s) 1] “Robot observes a person holding an object and walking towards a fridge (a). It uses our ATCRF to anticipate the affordances (b), and trajectories (c). It then performs an anticipatory action of opening the door (d).” [sec(s) 1] “our goal is to use anticipation for predicting future activities as well as improving detection (of past activities). … Therefore, we can anticipate the future by observing the sub-activities performed in the past and reasoning about the future based on the structure of activities and the functionality of objects being used (also referred to as object affordances [10]). For example, in Fig. 1, on seeing a person carrying a bowl and walking towards the refrigerator, one of the most likely future actions are to reach the refrigerator, open it and place the bowl inside. … In our work, we use a conditional random field based on [5] (see Fig. 2) to model the spatio-temporal structure of activities, as described in Section 5.1. For anticipation, we present an anticipatory temporal conditional random field (ATCRF), where we model the past with the CRF described above but augmented with the trajectories and with nodes/edges representing the object affordances, sub-activities, and trajectories in the future. Since there are many possible futures, each ATCRF represents only one of them. In order to find the most likely ones, we consider each ATCRF as a particle and propagate them over time, using the set of particles to represent the distribution over the future possible activities. One challenge is to use the discriminative power of the CRFs (where the observations are continuous and labels are discrete) for also producing the generative anticipation—labels over sub-activities, affordances, and spatial trajectories.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Lin, Kerzner with the inference of potential future event of Koppula.
One of ordinary skill in the art would have been motived to combine in order to improve activity anticipation performance for the future and improve present and past performance on detection based on the future anticipation.
(Koppula [sec(s) 1] “In this paper, our goal is to use anticipation for predicting future activities as well as improving detection (of past activities). … Our algorithm obtains an activity anticipation accuracy (defined as whether one of top three predictions actually happened) of (84.1, 74.4, 62.2 percent) for predicting (1, 3, 10) seconds into the future.” [sec(s) 8.3] “Our full model (row 6), which estimates the graph structure for both past and the future, improves the anticipation performance further. … This shows that anticipating the future can improve present and past performance on detection.”)
Regarding claim 18
The claim is a system claim corresponding to the method claim 8, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (Temporal image analytics for abnormal construction activity identification) in view of Kerzner et al. (US 11,195,311 B1) in view of Jungling et al. (Person re-identification in multi-camera networks)
Regarding claim 10
The combination of Lin, Kerzner teaches claim 1.
Lin teaches
detecting the particular activity comprises: (see the rejections of claim 1)
However, the combination of Lin, Kerzner does not appear to explicitly teach:
identifying a person associated with the particular activity across sensor data from a plurality of sensors.
Jungling teaches
identifying a person associated with the particular activity across sensor data from a plurality of sensors.
(Jungling [fig(s) 5-9] [sec(s) 4] “We evaluate our re-identification approach in the iLids scenario [13] which is a real-world multi-camera scenario. It contains sequences recorded at an airport in official hours, which means it contains realistic surveillance data and thus all difficulties that are accompanied by this for both, tracking and re-identification. For re-identification evaluation, we use the data of two cameras with disjoint views. Sample images of the two cameras are shown in figure 5. As one can see, many challenges arise for person re-identification here, especially when seeking an evaluation under realistic conditions and thus a system that can be applied to real-world scenarios. This means, that for re-identification, we cannot assume that tracking results are flawless. In real-world applications this is only rarely the case. Specifically under the challenging conditions here, where multiple persons move through the scene and occlude each other, assuming the output of a tracker to be perfect, like many other re-identification approaches do, is unsustainable. In addition, many challenges for re-identification itself arise here: (i) people are partially occluded by luggage which affects person appearance, (ii) cameras observe the scene from different viewpoints, (iii) environmental conditions like lighting differ significantly between cameras and (iv) there are differences in the color representation between the cameras (which in fact does not affect our SIFT based approach).”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Lin, Kerzner with the sensor data from multiple sensors of Jungling.
One of ordinary skill in the art would have been motived to combine in order to make more robust to inter-sensor variations, allow for very efficient re-identification since the computational cheap first stages can be used to reduce the amount of data, and make applicable in real applications.
(Jungling [sec(s) 1] “Moreover, not using color makes this approach more robust to inter-sensor variations like different color schemes (within one spectrum). (ii) The multi-stage approach with increasing computational cost allows for very efficient re-identification since the computational cheap first stages can be used to reduce the amount of data (candidate models from the database) that has to be considered on the last stage. (iii) Compared to most other state-of-the-art approaches like [2, 3], this approach is applicable in real applications since it is integrated with a detection and tracking strategy.”)
Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Miyanishi et al. (Generating an Event Timeline about Daily Activities from a Semantic Concept Stream) teaches generating a single timeline from semantic data.
Heilbron et al. (Fast Temporal Activity Proposals for Efficient Detection of Human Actions in Untrimmed Videos) teaches a time line of human actions especially in sports videos.
Shrestha et al. (US 20200202136 A1) teaches a unified timeline.
HILLELI et al. (US 2021/0099317 A1) teaches a timeline for a dialog among people.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEHWAN KIM whose telephone number is (571)270-7409. The examiner can normally be reached Mon - Thu 7:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached on (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SEHWAN KIM/Examiner, Art Unit 2129
Prosecution Insights