Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Claims 1-20 invoke § 112(f). Limitations in these claims that invoke § 112(f) are:
“first teacher/student machine learning subsystem” recited in claims 1, 3, 4, 5, and 20, which is coupled with functional language such as “being operative to teach said second student/teacher machine learning subsystem in a first instance” (recited in claims 1 and 20).
“second student/teacher machine learning subsystem” recited in claims 1, 3, 4, 5, and 20, which is coupled with functional language such as “being operative to teach said first teacher/student machine learning subsystem in a second instance” (recited in claims 1 and 20).
“additional student/teacher machine learning subsystem… being operative to teach said second student/teacher machine learning subsystem” recited in claim 3.
“Machine Learning Generative Module, operative to receive said data sensed by one of said first and second sensors and to generate” recited in claims 9, 10, 12, 13, 16, 17, and 18.
“a paired data provider operative to provide to provide” recited in claims 12 and 17.
“first generator sub-module operative to…generate…” recited in claims 13-14 and 17.
“second generator sub-module operative to…generate…” recited in claims 13-15, 17, and 19.
“additional generator sub-module operative to…generate…” recited in claims 15 and 19.
In the above terms, “subsystem”, “module,” “sub-module,” and “provider” are regarded as generic placeholder terms.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The following claim limitations invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function.
“first teacher/student machine learning subsystem” recited in claims 1, 3, 4, 5, and 20, which is coupled with functional language such as “being operative to teach said second student/teacher machine learning subsystem in a first instance” (recited in claims 1 and 20).
“second student/teacher machine learning subsystem” recited in claims 1, 3, 4, 5, and 20, which is coupled with functional language such as “being operative to teach said first teacher/student machine learning subsystem in a second instance” (recited in claims 1 and 20).
“additional student/teacher machine learning subsystem… being operative to teach said second student/teacher machine learning subsystem” recited in claim 3.
“Machine Learning Generative Module, operative to receive said data sensed by one of said first and second sensors and to generate” recited in claims 9, 10, 12, 13, 16, 17, and 18.
“a paired data provider operative to provide to provide” recited in claims 12 and 17.
“first generator sub-module operative to…generate…” recited in claims 13-14 and 17.
“second generator sub-module operative to…generate…” recited in claims 13-15, 17, and 19.
“additional generator sub-module operative to…generate…” recited in claims 15 and 19.
These limitations are indefinite because the specification does not teach a corresponding structure, such as a hardware computer or processor, that constitute the subsystem, module, submodule or provider quoted above. See MPEP § 2181(II)(B): “if there is no corresponding structure disclosed in the specification (i.e., the limitation is only supported by software and does not correspond to an algorithm and the computer or microprocessor programmed with the algorithm), the limitation should be deemed indefinite as discussed above, and the claim should be rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
The various subsystems and modules noted above are disclosed as software models in the specification. However, the hardware component for these limitations is not specified. Although the specification does teach a “processor,” the structural relationship between the “processor” and the above features are not linked to a processor.
Therefore, the claims that include one or more of the terms listed above or depend from a claim that includes one or more of the terms listed above is indefinite and are rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
1. Claims 1-2, 5-7, 9-10, 12, 16-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Aydogdu et al., "Multi-Modal Cross Learning for Improved People Counting using Short-Range FMCW Radar," 2020 IEEE International Radar Conference (RADAR), Washington, DC, USA, April 28-30, 2020, pp. 250-255 (“Aydogdu”) in view of Li et al., “Towards Cross-Modality Medical Image Segmentation with Online Mutual Knowledge Distillation,” The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20), February 7-12, 2020 (“Li”).
As to claim 1, Aydogdu teaches a sensing system comprising:
at least a first sensor sensing first data from a scene; [§ V, paragraph 1: “The data was collected by syncing four cameras mounted in four corners of a room of size 8m×10m and four radars placed just below them.” That is, the cameras correspond to a first sensor.]
at least a second sensor sensing second data from said scene; [§ V, paragraph 1: “The data was collected by syncing four cameras mounted in four corners of a room of size 8m×10m and four radars placed just below them.” That is, the radars correspond to a second sensor. Both the cameras and radar collect data from the same scene, which in this example is the room.]
a first teacher/student machine learning subsystem employable by said first sensor to process said first data; [FIG. 1, teaching a “camera DCNN” which is a teacher model in knowledge distillation. See § IV, paragraph 1: “The aim of the proposed training framework is to improve the learning process by transferring or distilling the high-level knowledge abstraction (density heatmap) from the camera modality to the radar DCNN, and acts as an additional supervision, much like the teacher-student network.”] and
a second student/teacher machine learning subsystem employable by said second sensor to process said second data, [FIG. 1, teaching a “radar DCNN” which is a student model in knowledge distillation. See § IV, paragraph 1: “The aim of the proposed training framework is to improve the learning process by transferring or distilling the high-level knowledge abstraction (density heatmap) from the camera modality to the radar DCNN, and acts as an additional supervision, much like the teacher-student network.”]
said first teacher/student machine learning subsystem being operative to teach said second student/teacher machine learning subsystem in a first instance, [§ IV, paragraph 1: “The aim of the proposed training framework is to improve the learning process by transferring or distilling the high-level knowledge abstraction (density heatmap) from the camera modality to the radar DCNN, and acts as an additional supervision, much like the teacher-student network.”]
Aydogdu does not explicitly teach “said second student/teacher machine learning subsystem being operative to teach said first teacher/student machine learning subsystem in a second instance.”
Li teaches “said second student/teacher machine learning subsystem being operative to teach said first teacher/student machine learning subsystem in a second instance” [FIG. 1, Sreal corresponds to a machine learning subsystem and Ssyn. As shown in this figure, Sreal and Ssyn both act as students and teachers for one another in a mutual distillation process. This is described in Algorithm 1, which teaches the losses in line 12, and the caption of FIG. 1: “The magenta-to-blue transition arrow represents the knowledge transfer from Ssyn to Sreal guided by Ls→rkd, while the blue-to-magenta transition arrow demonstrates the knowledge transfer from Sreal to Ssyn guided by Lr→skd.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Aydogdu with the teachings of Li by implementing the mutual knowledge distillation technique (see Li, title), such that both models train each other mutually, so as to arrive at the claimed invention. The motivation would have been to use a learning technique that enables exploit modality-shared knowledge to facilitate the target-modality segmentation (see Li, abstract: “We then propose a novel Mutual Knowledge Distillation (MKD) scheme to thoroughly exploit the modality-shared knowledge to facilitate the target-modality segmentation.”).
As to claim 2, the combination of Aydogdu and Li teaches the sensing system of claim 1, as set forth above.
Li also teaches wherein said first and second instances occur at least one of sequentially, partially concurrently and repeatedly over time. [Li, Algorithm 1, lines 12-13, which teaches that the update of the respective model parameters occur sequentially. Alternatively, the other option of “partially concurrently” is also met since the mutual distillation process as described here is also partially concurrent in that the losses or their components can be regarded as being calculated in a single process. Additionally, “repeatedly over time” is also taught because the process in Algorithm 1 repeats over multiple iterations.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of combined the teachings of Aydogdu with the teachings of Li so as to have also arrived at the claimed invention of the instant dependent claim. The motivation for doing so is covered by the motivation given for Li in the rejection of the parent claim.
As to claim 5, the combination of Aydogdu and Li teaches the sensing system of claim 1, wherein, in said first instance, said first teacher/student machine learning subsystem is operative to teach said second student/teacher machine learning subsystem to automatically label said second data [Aydogdu, § I, paragraph 3: “A learning system that involves a single modality data stream (e.g. radar) for the task of classification or regression is known as unimodal learning. Multi-modal learning on the other hand involves use of multiple sensor data streams (e.g. camera & radar) for learning to perform a common task of classification or regression and likewise the same sensor fusion setup for testing or inferring in real-time…. Fig. 1 presents the high-level concept of the proposed multi-modal cross learning framework where radar DCNN learns not only from radar data but also supervised representation from camera DCNN trained on similar task.”] […] said first and second data being mutually calibrated with respect to one another in each of said first and second instances. [Aydogdu, § V, paragraph 1: “The data was collected by syncing four cameras mounted in four corners of a room of size 8m×10m and four radars placed just below them.” That is, placing the sensors in the same room constitutes calibrating them to the same scene.]
Li further teaches “in said second instance, said second student/teacher machine learning subsystem is operative to teach said first teacher/student machine learning subsystem to automatically label said first data.” [Caption of FIG. 1: “The magenta-to-blue transition arrow represents the knowledge transfer from Ssyn to Sreal guided by Ls→rkd, while the blue-to-magenta transition arrow demonstrates the knowledge transfer from Sreal to Ssyn guided by Lr→skd.” Note that this knowledge transfer is for labeling data, as characterized on page 778 (section titled “Mutual Knowledge Distillation”) and shown in FIG. 1 which refers to the labels yt and ya as respective outputs of the model.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of combined the teachings of Aydogdu with the teachings of Li so as to have also arrived at the claimed invention of the instant dependent claim. The motivation for doing so is covered by the motivation given for Li in the rejection of the parent claim.
As to claim 6, the combination of Aydogdu and Li teaches the sensing system of claim 1, wherein said at least first and second sensors comprise mutually different types of sensors. [Aydogdu, abstract: “Radar systems enable remote-less sensing of multiple persons in its field of view. In this paper, we propose a novel people counting system using 60-GHz frequency modulated continuous wave radar sensor. The proposed deep convolutional neural network learns from supervised radar data and also through knowledge distillation via multi-modal cross-learning of representation from a synchronized camera-based deep convolutional neural network.” § V, paragraph 1: “The data was collected by syncing four cameras mounted in four corners of a room of size 8m×10m and four radars placed just below them.”]
As to claim 7, the combination of Aydogdu and Li teaches the sensing system of claim 6, wherein said at least first and second sensors comprise at least one of the following: one of said first and second sensors is a camera and the other one of said first and second sensors is an active or passive radar; one of said first and second sensors is an active radar and the other one of said first and second sensors is a passive radar; and one of said first and second sensors is an ultrasound sensor and the other one of said first and second sensors is an ECG sensor. [Aydogdu, abstract: “Radar systems enable remote-less sensing of multiple persons in its field of view. In this paper, we propose a novel people counting system using 60-GHz frequency modulated continuous wave radar sensor. The proposed deep convolutional neural network learns from supervised radar data and also through knowledge distillation via multi-modal cross-learning of representation from a synchronized camera-based deep convolutional neural network.” § V, paragraph 1: “The data was collected by syncing four cameras mounted in four corners of a room of size 8m×10m and four radars placed just below them.” That is, noting that the instant claim recites an alternate expression, this reference teaches the alternative of “one of said first and second sensors is an active radar and the other one of said first and second sensors is a passive radar.” Note that the radar used here, BGT60TR13C, is an active radar.]
As to claim 9, the combination of Aydogdu and Li teaches the sensing system of claim 1, as set forth above.
Li further teaches “also comprising a Machine Learning Generative Module, operative to receive said data sensed by one of said first and second sensors and to generate, using machine learning and based on said received data, a generated representation of said scene corresponding to data sensed by the other one of said first and second sensors.” [FIG. 1 caption: “The generator Ga→t performs assistant-to-target translation and outputs synthesized CT data.” Page 777, “Image Alignment Module” section, paragraph 1: “Since redundant modality-specific appearances would introduce bias and increase the difficulty in leveraging the valuable modality-shared prior knowledge from Xa, we propose to perform a transformation from assistant-modality image xa to target-modality image xt. In this way, the synthetic target-modality image xa→t would acquire similar appearances to target-modality data with unaffected assistant modality structures.” That is, as shown in FIG. 1, where an MRI image is converted, by synthesis, into a CT image using the generator.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of combined the teachings of Aydogdu with the teachings of Li so as to have also arrived at the claimed invention of the instant dependent claim. The motivation for doing so is covered by the motivation given for Li in the rejection of the parent independent claim.
As to claim 10, the combination of Aydogdu and Li teaches the sensing system according to claim 9, as set forth above.
Li further teaches “wherein said Machine Learning Generative Module comprises a Generative Adversarial Network (GAN).” [Page 777, “Image Alignment Module” section: “Inspired by generative adversarial networks (Goodfellow et al. 2014), we adopt a generator Ga→t and discriminator Dt to perform the assistant-to-target image translation Ga→t : Xa → Xt by adversarial learning.” Note that while this part of the text only refers to “inspired” by GAN, it is clear that the model as described in this section is a GAN, based on equations (1) and (2).]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of combined the teachings of Aydogdu with the teachings of Li so as to have also arrived at the claimed invention of the instant dependent claim. The motivation for doing so is covered by the motivation given for Li in the rejection of the parent independent claim.
As to claim 12, the combination of Aydogdu and Li teaches the sensing system of claim 9, as set forth above.
Li further teaches wherein said Machine Learning Generative Module comprises:
a generator sub-module operative to receive said data sensed by said one of said first and second sensors and to generate, using machine learning and based on said data sensed by said one of said first and second sensors, said representation of said scene corresponding to said data sensed by said other one of said first and second sensors; [FIG. 1 caption: “The generator Ga→t performs assistant-to-target translation and outputs synthesized CT data.” Page 777, “Image Alignment Module” section, paragraph 1: “Since redundant modality-specific appearances would introduce bias and increase the difficulty in leveraging the valuable modality-shared prior knowledge from Xa, we propose to perform a transformation from assistant-modality image xa to target-modality image xt. In this way, the synthetic target-modality image xa→t would acquire similar appearances to target-modality data with unaffected assistant modality structures.” That is, as shown in FIG. 1, where an MRI image is converted, by synthesis, into a CT image using the generator. This generated CT image constitutes a representation of said scene corresponding to data sensed by the other sensor (MRI). Furthermore, the limitation of “receive said data” is met because the generator in Li uses data that was generated by CT.] and
a paired data provider operative to provide to said generator sub-module pairs of mutually corresponding data previously sensed from said scene by said first and second sensors, said generator sub-module being operative to take into account said pairs of mutually corresponding previously sensed data in generating said representation of said scene. [Page 777, “Methodology” section: “Given a set of labeled samples {xai, yai}Ni=1 from assistant-modality data Xa, and a set of labeled samples {xtj, ytj}Mj=1 from target modality data Xt, we involve both Xa and Xt in network training, to improve the segmentation performance on target modality during testing.” As shown in FIG. 1, the specific images xa and xt from the respective sets are provided as a pair to the system. Note that the limitation of “provider…to provide” is met because it is implied that the operations are performed on a computer.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of combined the teachings of Aydogdu with the teachings of Li so as to have also arrived at the claimed invention of the instant dependent claim. The motivation for doing so is covered by the motivation given for Li in the rejection of the parent independent claim.
As to claim 16, the combination of Aydogdu and Li teaches the sensing system of claim 9, wherein said Machine Learning Generative Module is operative to synthesise labelled training data useful for the training of at least one of said first and second machine learning subsystems.” [FIG. 1 caption: “The generator Ga→t performs assistant-to-target translation and outputs synthesized CT data.” Page 777, “Image Alignment Module” section, paragraph 1: “Since redundant modality-specific appearances would introduce bias and increase the difficulty in leveraging the valuable modality-shared prior knowledge from Xa, we propose to perform a transformation from assistant-modality image xa to target-modality image xt. In this way, the synthetic target-modality image xa→t would acquire similar appearances to target-modality data with unaffected assistant modality structures.” That is, as shown in FIG. 1, where an MRI image is converted, by synthesis, into a CT image using the generator. This constitutes generating a synthesized image. Furthermore, this image is labeled, as disclosed on page 777, “Methodology’ section first paragraph, which teaches that the samples from the assistant modality are labeled yai. This label applies to the generated image xa→t because xa→t is corresponding to xa, and the original label for xa is used in equation (3).]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of combined the teachings of Aydogdu with the teachings of Li so as to have also arrived at the claimed invention of the instant dependent claim. The motivation for doing so is covered by the motivation given for Li in the rejection of the parent claim.
As to claim 17, Aydogdu teaches a sensing system comprising:
at least a first sensor sensing first data from a scene; [§ V, paragraph 1: “The data was collected by syncing four cameras mounted in four corners of a room of size 8m×10m and four radars placed just below them.” That is, the cameras correspond to a first sensor.]
at least a second sensor sensing second data from said scene, said first data being of a different type than said second data; [§ V, paragraph 1: “The data was collected by syncing four cameras mounted in four corners of a room of size 8m×10m and four radars placed just below them.” That is, the radars correspond to a second sensor. Both the cameras and radar collect data from the same scene, which in this example is the room.].
Aydogdu does not explicitly teach:
a Machine Learning Generative Module comprising at least one of:
(i) a generator sub-module operative to receive said first data sensed from said scene and to generate, using machine learning and based on said first data, a representation of said scene corresponding to said second type of data, and a paired data provider operative to provide to said generator sub-module pairs of mutually corresponding first type of data and second type of data previously sensed from said scene, said generator sub-module being operative to take into account said pairs of mutually corresponding previously sensed first and second types of data in generating said representation of said scene, and
(ii) a first generator sub-module operative to receive said first data and to generate, using machine learning and based on said first data, a representation of said scene corresponding to said second type of data, and a second generator sub-module operative to receive said representation of said scene generated by said first generator sub-module and said first data and to generate, using machine learning, a refined representation of said scene corresponding to said second type of data, based on said representation of said scene generated by said first generator sub-module and said first data.
Li teaches “a Machine Learning Generative Module comprising at least one of: (i) a generator sub-module operative to receive said first data sensed from said scene and to generate, using machine learning and based on said first data, a representation of said scene corresponding to said second type of data,” [FIG. 1 caption: “The generator Ga→t performs assistant-to-target translation and outputs synthesized CT data.” Page 777, “Image Alignment Module” section, paragraph 1: “Since redundant modality-specific appearances would introduce bias and increase the difficulty in leveraging the valuable modality-shared prior knowledge from Xa, we propose to perform a transformation from assistant-modality image xa to target-modality image xt. In this way, the synthetic target-modality image xa→t would acquire similar appearances to target-modality data with unaffected assistant modality structures.” That is, as shown in FIG. 1, where an MRI image is converted, by synthesis, into a CT image using the generator. This generated CT image constitutes a representation of said scene corresponding to data sensed by the other sensor (MRI). Furthermore, the limitation of “receive said data” is met because the generator in Li uses data that was generated by CT.] “and a paired data provider operative to provide to said generator sub-module pairs of mutually corresponding first type of data and second type of data previously sensed from said scene, said generator sub-module being operative to take into account said pairs of mutually corresponding previously sensed first and second types of data in generating said representation of said scene” [Page 777, “Methodology” section: “Given a set of labeled samples {xai, yai}Ni=1 from assistant-modality data Xa, and a set of labeled samples {xtj, ytj}Mj=1 from target modality data Xt, we involve both Xa and Xt in network training, to improve the segmentation performance on target modality during testing.” As shown in FIG. 1, the specific images xa and xt from the respective sets are provided as a pair to the system. Note that the limitation of “provider…to provide” is met because it is implied that the operations are performed on a computer.]
Note: This part of the claim recites an alternative expression that is met when either (i) or (ii) is met. Here, Li is relied upon below to teach alternative (i) fully. Thus, for purposes of the rejection of this claim, item (ii) does not need to be met. However, the parts of Li discussed above would also meet the limitation of “a first generator sub-module operative to receive said first data and to generate, using machine learning and based on said first data, a representation of said scene corresponding to said second type of data.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Aydogdu with the teachings of Li by implementing the mutual knowledge distillation technique (see Li, title) including the use of a generator as discussed above, such that both models train each other mutually and that the model for the first data is used with a generator, so as to arrive at the claimed invention. The motivation would have been to use a learning technique that enables exploit modality-shared knowledge to facilitate the target-modality segmentation (see Li, abstract: “We then propose a novel Mutual Knowledge Distillation (MKD) scheme to thoroughly exploit the modality-shared knowledge to facilitate the target-modality segmentation.”).
As to claim 18, the combination of Aydogdu and Li teaches the sensing system according to claim 17, as set forth above.
Li further teaches “wherein said Machine Learning Generative Module comprises a Generative Adversarial Network (GAN).” [Page 777, “Image Alignment Module” section: “Inspired by generative adversarial networks (Goodfellow et al. 2014), we adopt a generator Ga→t and discriminator Dt to perform the assistant-to-target image translation Ga→t : Xa → Xt by adversarial learning.” Note that while this part of the text only refers to “inspired” by GAN, it is clear that the model as described in this section is a GAN, based on equations (1) and (2).]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of combined the teachings of Aydogdu with the teachings of Li so as to have also arrived at the claimed invention of the instant dependent claim. The motivation for doing so is covered by the motivation given for Li in the rejection of the parent independent claim.
As to claim 20, the combination of Aydogdu and Li teaches the sensing system of claim 17 and also comprising:
a first teacher/student machine learning subsystem employable by one of said first and second sensors to process said data sensed thereby; [FIG. 1, teaching a “camera DCNN” which is a teacher model in knowledge distillation. See § IV, paragraph 1: “The aim of the proposed training framework is to improve the learning process by transferring or distilling the high-level knowledge abstraction (density heatmap) from the camera modality to the radar DCNN, and acts as an additional supervision, much like the teacher-student network.”] and
a second student/teacher machine learning subsystem employable by the other one of said first and second sensors to process said data sensed thereby, [FIG. 1, teaching a “radar DCNN” which is a student model in knowledge distillation. See § IV, paragraph 1: “The aim of the proposed training framework is to improve the learning process by transferring or distilling the high-level knowledge abstraction (density heatmap) from the camera modality to the radar DCNN, and acts as an additional supervision, much like the teacher-student network.”]
said first teacher/student machine learning subsystem being operative to teach said second student/teacher machine learning subsystem in a first instance, […] [§ IV, paragraph 1: “The aim of the proposed training framework is to improve the learning process by transferring or distilling the high-level knowledge abstraction (density heatmap) from the camera modality to the radar DCNN, and acts as an additional supervision, much like the teacher-student network.”]
Aydogdu does not explicitly teach “said second student/teacher machine learning subsystem being operative to teach said first teacher/student machine learning subsystem in a second instance.”
Li teaches “said second student/teacher machine learning subsystem being operative to teach said first teacher/student machine learning subsystem in a second instance” [FIG. 1, Sreal corresponds to a machine learning subsystem and Ssyn. As shown in this figure, Sreal and Ssyn both act as students and teachers for one another in a mutual distillation process. This is described in Algorithm 1, which teaches the losses in line 12, and the caption of FIG. 1: “The magenta-to-blue transition arrow represents the knowledge transfer from Ssyn to Sreal guided by Ls→rkd, while the blue-to-magenta transition arrow demonstrates the knowledge transfer from Sreal to Ssyn guided by Lr→skd.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of combined the teachings of Aydogdu with the teachings of Li so as to have also arrived at the claimed invention of the instant dependent claim. The motivation for doing so is covered by the motivation given for Li in the rejection of the parent claim.
2. Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Aydogdu in view of Li, and further in view of Guo et al., “Online Knowledge Distillation via Collaborative Learning,” CVPR 2020, June 14-19, 2020 (“Guo”)
As to claim 3, the combination of Aydogdu and Li teaches the sensing system of claim 1, but does not teach the further limitations of the instant dependent claim.
Guo teaches “also comprising at least one additional student/teacher machine learning subsystem, said first teacher/student machine learning subsystem being operative to teach said at least one additional student/teacher machine learning subsystem in a third instance, said at least one additional student/teacher machine learning subsystem being operative to teach said second student/teacher machine learning subsystem in a fourth instance.” [As shown in FIG. 2, there are m models all participating in a knowledge distillation process, such that all models are teachers and students. See FIG. 2 caption: “Figure 2: Overview of knowledge distillation via collaborative learning (KDCL). We input images distorted separately for each network to increase the invariance against perturbations in the data domain. KDCL dynamically ensembles soft target produced by all students to improve students consistently. h(x, ǫ) means random distortion and ǫ is the random seed.” See also equation (3), which teaches the teacher logit zt, which is a function of the m sub-networks, and the text above: “In our framework, all the models are student models and the supervision is generated by combining the output of the models.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references thus far with the teachings of Guo so as to arrive at the claimed invention of the instant dependent claim. The motivation would have been to collaborative train arbitrary number of networks in a manner that enables penalization and generalization ability (see Guo, abstract: “Unlike existing two stage knowledge distillation approaches that pre-train a DNN with large capacity as the “teacher” and then transfer the teacher’s knowledge to another “student” DNN unidirectionally (i.e. one-way), KDCL treats all DNNs as “students” and collaboratively trains them in a single stage (knowledge is transferred among arbitrary students during collaborative training), enabling parallel computing, fast computations, and appealing generalization ability”).
3. Claim 4 are rejected under 35 U.S.C. 103 as being unpatentable over Aydogdu in view of Li, and further in view of Li et al. (US 2019/0051290 A1) (“Li (US ‘290)”).
As to claim 4, the combination of Aydogdu and Li teaches the sensing system of claim 1, wherein, upon said second student/teacher machine learning subsystem […], said first sensor is deactivated and said first teacher/student machine learning subsystem is operative to stop teaching said second student/teacher machine learning subsystem. [As shown in Aydogdu, FIG. 1, after the training is completed, only the radar data is used. See § V, paragraph 2: “During inference, we just need the trained encoder block with the classification block which takes in input the raw radar data and predicts the count class. The entire inference block has a very small memory footprint of 44 kB, which is an added advantage arising from the proposed multi-modal cross learning.” Therefore, it is implied that the camera is deactivated, at least for purposes of use of the system, after training is completed.]
The combination of references thus far does not use “achieving a pre-determined performance” as a condition for completing the training (the Examiner notes that Li teaches “converge” as a stopping condition, but is not explicit as to how this relates to pre-determined performance).
Li (US ‘290) teaches “achieving a pre-determined performance” [[0043]: “In successive iterations of training the student model 160 the successive parallel batches will be fed to the teacher model 150 and the student model 160 to produce successive posteriors, which will be compared again against one another until a maximum number of epochs is reached, the divergence score satisfies a convergence threshold, divergence plateaus, or training is manually stopped.” See also [0037]-[0038], which teaches that the divergence score is a performance metric.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of Li (US ‘290) by “achieving a pre-determined performance” as a condition for completing the training, so as to arrive at the claimed invention of the instant dependent claim. Doing so would have been enabled the determination of convergence for a training process, as suggested by Li (US ‘290) (see parts quoted above).
4. Claim 8 are rejected under 35 U.S.C. 103 as being unpatentable over Aydogdu in view of Li, and further in view of Sohn et al. (US 2018/0268266 A1) (“Sohn”).
As to claim 8, the combination of Aydogdu and Li teaches the sensing system of claim 1, but does not explicitly teach “wherein at least one of said at least first and second sensors is a remote sensor.”
Sohn teaches “wherein at least one of said at least first and second sensors is a remote sensor.” [[0031]: “The surveillance can involve detecting the presence of objects, recognizing the objects, identifying particular actions performed by the objects, and/or performing one or more actions (e.g., in response to object recognition/surveillance results). The server 120 can be located remote from, or proximate to, the camera system 110. The server 120 can include a processor 121, a memory 122, and a wireless transceiver 123. The processor 121 and the memory 122 of the remote server 120 can be configured to perform surveillance based on images received from the camera system 110 by the (the wireless transceiver 123 of) the remote server 120.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of Sohn by implementing the remote application taught in Sohn. Doing so would have enabled remote surveillance in which processing can be performed by a remote server, as suggested by Sohn (see parts quoted above).
5. Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Aydogdu in view of Li, and further in view of Isola et al., “Image-to-Image Translation with Conditional Adversarial Networks,” CVPR 2017 (“Isola”).
As to claim 11, the combination of Aydogdu and Li teaches the sensing system according to claim 10, as set forth above, but does not teach “wherein said GAN is a conditional GAN.”
Isola teaches “wherein said GAN is a conditional GAN.” [Abstract: “We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of Isola by implementing the GAN as a conditional GAN. The motivation would have been to use a specific type of GAN that is effective in synthesizing images (see parts quoted above), “especially those involving highly structured graphical outputs” (Isola, § 5).
6. Claims 13-15, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Aydogdu in view of Li, and further in view of Vo et al., “Visual-Relation Conscious Image Generation from Structured-Text,” arXiv:1908.01741v2 [cs.CV] 27 Nov 2019
As to claim 13, the combination of Aydogdu and Li teaches the sensing system of claim 9, wherein said Machine Learning Generative Module comprises:
a first generator sub-module operative to receive said data sensed by said one of said first and second sensors and to generate, using machine learning and based on said data sensed by said one of said first and second sensors, said representation of said scene corresponding to said data sensed by said other one of said first and second sensors; [FIG. 1 caption: “The generator Ga→t performs assistant-to-target translation and outputs synthesized CT data.” Page 777, “Image Alignment Module” section, paragraph 1: “Since redundant modality-specific appearances would introduce bias and increase the difficulty in leveraging the valuable modality-shared prior knowledge from Xa, we propose to perform a transformation from assistant-modality image xa to target-modality image xt. In this way, the synthetic target-modality image xa→t would acquire similar appearances to target-modality data with unaffected assistant modality structures.” That is, as shown in FIG. 1, where an MRI image is converted, by synthesis, into a CT image using the generator. This generated CT image constitutes a representation of said scene corresponding to data sensed by the other sensor (MRI). Furthermore, the limitation of “receive said data” is met because the generator in Li uses data that was generated by CT.]
The combination of references thus far does not teach the remaining limitations of this claim.
Vo teaches “a second generator sub-module operative to receive said representation of said scene generated by said first generator sub-module and said data sensed by said one of said first and second sensors and to generate, using machine learning, a generated refined representation of said scene corresponding to said data sensed by said other one of said first and second sensors, based on said representation of said scene generated by said first generator sub-module and said data sensed by said one of said first and second sensors.” [FIG. 2, which teaches generators G2 and G3, which are successive generators (generator submodules). See FIG. 2 caption: “The second (the third) GAN receives the upsampled visual-relation layout and the hidden features of previous GAN as its inputs.” See § 3.2: “We employ a stacking-GANs to progressively generate coarse-to-fine images. It consists of three GANs, each of which is conditioned on θ(t) (Fig. 4). The generator is built upon five refinement layers …The first GAN generator receives the visual-relation lay out θ(t) and a standard Gaussian distribution noise as input while the others receive the bilinear upsampled [21] layout θ(t) and the output of the last refinement layer from the previous GAN.” That is, the progressive coarse-to-fine images are successive refinements after the first GAN, and each subsequent refinement is based on the inputs to the previous GAN, analogous to the “based on” limitation of the instant claim.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of Vo so as to arrive at the claimed invention of the instant dependent claim. The motivation would have been to images with realism (see Vo, § 3.3, second-to-last paragraph: “Adversarial loss [4] Ladv is used to encourage the stacking-GANs to generate realistic images.”) by generating of coarse-to-fine images progressively to preserve scene structure (see Vo, § 1, paragraph 4: “progressively generating coarse-to-fine images with the pyramid of GANs, namely stacking-GANs, conditioned on the visual-relation layout”; § 2, paragraph 4: “Recursively conditioning stacking-GANs on our constructed visual-relation layout enables us to progressively generate coarse-to-fine images that consistently preserve the scene structure”).
As to claim 14, the combination of Aydogdu, Li, and Vo teaches the sensing system of claim 13, as set forth above.
Vo further teaches “wherein said refined representation of said scene generated by said second generator sub-module is newly generated with respect to said representation of said scene generated by said first generator sub-module.” [This limitation is met by the stacked arrangement of generates in Vo because each successive generator as taught in Vo generates an output, which is “newly generated” in that it did not exist previously.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of combined the teachings of the references combined thus far, including those of Vo discussed above for the instant claim, so as to have also arrived at the claimed invention of the instant dependent claim. The motivation for doing so is covered by the motivation given for Li in the rejection of the parent independent claim.
As to claim 15, the combination of Aydogdu, Li, and Vo teaches the sensing system of claim 13, as set forth above.
Vo further teaches “and also comprising at least one additional generator sub-module operative to receive said refined representation of said scene generated by said second generator sub-module and said data sensed by said one of said first and second sensors and to generate, using machine learning, a further refined representation of said scene corresponding to said data sensed by said other one of said first and second sensors, based on said refined representation of said scene generated by said second generator sub-module and said data sensed by said one of said first and second sensors.” [FIG. 2, which teaches generators G2 and G3, which are successive generators (generator submodules). See FIG. 2 caption: “The second (the third) GAN receives the upsampled visual-relation layout and the hidden features of previous GAN as its inputs.” See § 3.2: “We employ a stacking-GANs to progressively generate coarse-to-fine images. It consists of three GANs, each of which is conditioned on θ(t) (Fig. 4). The generator is built upon five refinement layers …The first GAN generator receives the visual-relation lay out θ(t) and a standard Gaussian distribution noise as input while the others receive the bilinear upsampled [21] layout θ(t) and the output of the last refinement layer from the previous GAN.” That is, the progressive coarse-to-fine images are successive refinements after the first GAN, and the refinement from the third GAN (G2) is based on the inputs to the previous GAN, analogous to the “based on” limitation of the instant claim.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of combined the teachings of the references combined thus far, including those of Vo discussed above for the instant claim, so as to have also arrived at the claimed invention of the instant dependent claim. The motivation for doing so is covered by the motivation given for Li in the rejection of the parent independent claim.
As to claim 19, the combination of Aydogdu and Li teaches the sensing system of claim 17, but does not teach the further limitations of the instant dependent claim.
Vo teaches “and also comprising at least one additional generator sub-module operative to receive said refined representation of said scene generated by said second generator sub-module and said first data and to generate, using machine learning, a further refined representation of said scene corresponding to said second type of data, based on said refined representation of said scene generated by said second generator sub-module and said first data.” [FIG. 2, which teaches generators G2 and G3, which are successive generators (generator submodules). See FIG. 2 caption: “The second (the third) GAN receives the upsampled visual-relation layout and the hidden features of previous GAN as its inputs.” See § 3.2: “We employ a stacking-GANs to progressively generate coarse-to-fine images. It consists of three GANs, each of which is conditioned on θ(t) (Fig. 4). The generator is built upon five refinement layers …The first GAN generator receives the visual-relation lay out θ(t) and a standard Gaussian distribution noise as input while the others receive the bilinear upsampled [21] layout θ(t) and the output of the last refinement layer from the previous GAN.” That is, the progressive coarse-to-fine images are successive refinements after the first GAN, and the refinement from the third GAN (G2) is based on the inputs to the previous GAN, analogous to the “based on” limitation of the instant claim. Note: To the extent that this dependent claim requires item (ii) of the alternative expression in claim 17 the “second generator sub-module” is met by Vo since Vo teaches a successive sequence of GANS.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of Vo so as to arrive at the claimed invention of the instant dependent claim. The motivation would have been to images with realism (see Vo, § 3.3, second-to-last paragraph: “Adversarial loss [4] Ladv is used to encourage the stacking-GANs to generate realistic images.”) by generating of coarse-to-fine images progressively to preserve scene structure (see Vo, § 1, paragraph 4: “progressively generating coarse-to-fine images with the pyramid of GANs, namely stacking-GANs, conditioned on the visual-relation layout”; § 2, paragraph 4: “Recursively conditioning stacking-GANs on our constructed visual-relation layout enables us to progressively generate coarse-to-fine images that consistently preserve the scene structure”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The following document depicts the state of the art.
Gupta et al., “Cross Modal Distillation for Supervision Transfer,” CVPR 2016, teaches knowledge distillation using multimodal inputs.
Thoker et al., “Cross-Modal Knowledge Distillation for Action Recognition,” ICIP 2019 teaches knowledge distillation using multimodal inputs.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YAO DAVID HUANG whose telephone number is (571)270-1764. The examiner can normally be reached Monday - Friday 9:00 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Y.D.H./Examiner, Art Unit 2124
/MIRANDA M HUANG/Supervisory Patent Examiner, Art Unit 2124
Prosecution Insights