DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
the two or more imaging modules (Applicant’s specification [0059] a first imaging module 102a and a second imaging module 102b) in claims 1,2,8,10, 20.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1,3,4,7,11,13,14 and 17 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Ben Gigi (US 11580838 B1, herein now referred as Ben).
Claim 1. Ben teaches a water surveillance system (Figs. 1A,1B) comprising:
two or more imaging modules, wherein each of the two or more imaging modules is configured to provide image data of an environment comprising a body of water
(Col 9 lines 30-40 system 100 may include two or more camera units 110, wherein camera 112 of each of two or more camera units 110 may be directly connected to main control unit 120.);
a processing unit communicatively connected to the two or more imaging modules
(Col 9 65-67 Col 10 lines 1-10 Main control unit 120 may...includes a main processing unit such as e.g., central processing unit (CPU),...artificial intelligence
system 100 may include two or more camera units 110, wherein camera 112 of each of two or more camera units 110 may be directly connected to main control unit 120.);
and a memory communicatively connected to the processing unit, wherein the memory comprises instructions configuring the processing unit to:
(Col 10 lines 1-10 Main control unit 120 may, for example, include a primary memory (e.g., RAM) and a secondary memory)
receive the image data of the environment from each of the two or more imaging modules
(Col 11 lines 60-67 (58) The method may include receiving 204, by a main control unit, the plurality of images from the at least one camera (e.g., main control unit 120 as described above with respect to FIGS. 1A, 1B and 1C).);
identify, using a neural network, one or more objects within the environment based on the image data, wherein identifying the one or more objects comprises associating an object identifier with each of the one or more objects
(Col 12 lines 20-25 The method may include detecting 208, by the main control unit, in an image of the plurality of images, one or more human bodies (e.g., as described below with respect to FIGS. 4A, 4B, 4C and 4D).
Col 14 lines 1-5, 40-50 pre-trained artificial intelligence (AI) body parts detection model. For example, the AI body parts detection model 420 may, for example, include a neural network 422,...The method may include defining 438, by the main control unit, two or more body part bounding boxes, each body part bounding box bounds one of the two or more detected body parts. (e.g. bounding boxes identifying human));
determine status data of the one or more objects based on the image data
(Col 12 35-40 The method may include determining 212, by the main control unit, for each of the one or more human bodies, based on a first subset of images of the plurality of images, based on the tracking of the respective human body and based on a predefined set of rules, whether or not the respective human body is suspicious of undergoing a pre-drowning event or a drowning event.);
provide a distress parameter based on the object identifier
(Col 18 lines 1-40 (121) The human body may be suspicious of undergoing the pre-drowning event according to some embodiments of the invention if: i) the head (represented in FIG. 7A by head bounding box 712a) of the human body in images 710 is below virtual safety line 714, (ii) an orientation of the human body (represented in FIG. 7A by body bounding box 712) in images 710 is vertical (or substantially vertical), (iii) a measure of motion of the human body in a vertical direction (represented in FIG. 7A by an arrow 715a) in images 710 is greater than a measure of motion of the human body in a horizontal direction (represented in FIG. 7A by an arrow 715b) in images 710, and (iv) a total measure of motion of the human body in images 710 is below a predefined motion threshold.
e.g. I,ii,iii,iv are distressed parameters);
determine a critical event related to at least one of the objects of the one or more objects based on the status data and a distress parameter
(Col 18 lines 15-60 Upon determination that the human body is suspicious of undergoing the pre-drowning event, the respective human body may be further tracked, by the main control unit... Upon determination that the human body is suspicious of undergoing the drowning event, the respective human body may be further tracked, by the main control unit, in an additional subset of subsequent images corresponding to a predefined time interval (for example, 5-15 seconds, e.g., 10 seconds). );
and generate an alert based on the determination of the critical event
Col 18 lines 15-65 the main control unit may determine that the human body is actually undergoing the pre-drowning event and may cause the alarm unit to issue an alarm....the main control unit may determine that the human body is actually undergoing the drowning event and may cause the alarm unit to issue an alarm.).
Claim 3. Ben teaches the system of claim 1, further comprising a display having a user interface providing Graphics
(Col 21 lines 1-10 (148) Input devices 935 may be or may include a mouse, a keyboard, a touch screen or pad or any suitable input device. It will be recognized that any suitable number of input devices may be operatively connected to computing device 900 as shown by block 935. Output devices 940 may include one or more displays, speakers and/or any other suitable output devices.),
wherein: determining the status data of the one or more objects comprises tracking, using a
tracking algorithm, a movement of the one or more objects within the environment
(Col 4 lines 5-10 based on the tracking of the respective human body and a predefined set of rules, whether or not the respective human body is suspicious of undergoing a pre-drowning event or a drowning event;
Col 9 lines 50-60 Main control unit 120 may determine, based on the received images and a predetermined set of rules, whether or not at least, one of the one or more human bodies undergoes a pre-drowning or a drowning event (e.g., as described below with respect to FIG. 2 and FIGS. 7A, 7B, 7C, 7D, 7E).);
determining the critical event further comprises comparing the movement of the one or
more objects to a threshold of the distress parameter
(Col 15 lines 55-65 (i) the respective body bounding boxes in the respective two subsequent images have a maximal overlap with each other as compared to overlaps between other body bounding boxes in the images and (ii) an overlap between the respective body bounding boxes in the respective two subsequent images is above a specified threshold (e.g., as schematically shown in FIG. 5B). Step 510 may be done using, for example, intersection over union (IoU) method (e.g., as schematically shown in FIG. 5B). Setting the minimal specified threshold (e.g., 5-10%) );
the tracking changes a corresponding weight of each level of a cascade matching algorithm based on a current situation, a status of the system, and/or detection characteristics
(Col 16 lines 60-67 Col 17 lines 1-25 e.g. sequence of body parts are detected (cascade matching algorithm)
Col 18 lines 40-50 (i) the orientation of the human body (represented in FIG. 7B by bounding box 722) in images 710 is horizontal (or substantially horizontal); (ii) the human body (represented in FIG. 7B by hounding box 722) in images 710 is below virtual safety line 724; and (iii) a total measure of motion of the human body (represented in FIG. 7A by an arrow 725a) in images 720 is below a predefined motion threshold.);
and the processing unit is further configured to render, on the display, a three-dimensional
(3D) tracking view on the display comprising visual indicators for each of the one or more
objects in the environment based on the tracking and/or object identifiers of the one or more
objects
(Col 5 lines 25-40 define, a first body part bounding box that bounds the first body part and a second body part bounding box that bounds the second body part in the image; calculate a first centroid point of the first body part bounding box and a second centroid point of the second body part bounding box in the image; determine an angle between (i) a line extending between the first centroid point and the second centroid point, and (ii) a virtual horizontal line in the image; and determine, based on the determined angle, an orientation of the human body in the image.
Col 17 lines 60-65 FIG. 7A schematically shows a subset 710 of subsequent underwater images 710a, 710b, 710c including a detected human body represented by a body bounding box 712. (e.g. 3D image shown with object identifiers)).
Claim 4. Ben teaches the system of claim 1, wherein: the status data comprises a current location of the one or more objects and/or a submergence of the one or more objects
(Col 4 lines 1-5 a position and orientation of the respective human body with respect to the virtual safety line and a measure of motion of the respective human body);
the distress parameter comprises a predetermined duration of time for submersion beneath a surface of the body of water
(Col 18 lines 15-25 in an additional subset of subsequent images corresponding to a predefined time interval (for example, 5-15 seconds, e.g., 10 seconds).);
and determining the critical event comprises determining that a first object of the one or more objects has been submerged beneath the surface of the body of water for more than the predetermined duration of time
(Col lines (i) the head (represented in FIG. 7A by head bounding box 712a) of the human body in images 710 is below virtual safety line 714,.. an additional subset of subsequent images corresponding to a predefined time interval (for example, 5-15 seconds, e.g., 10 seconds).).
Claim 7. Ben teaches the system of claim 1, wherein: the critical event comprises an object of the one or more objects being underwater for too long, approaching an edge of the body of water as a non-swimmer, running on wet pavement, and/or jumping or diving on another object
(Col 17 lines 55-60 Reference is now made to FIG. 7A, which schematically shows a subset 710 of subsequent underwater images including a human body that is suspicious of undergoing a pre-drowning event, according to some embodiments of the invention.);
and determining the critical event comprises identifying a level of the critical event;
(Col 18 lines 15-25 Upon determination that the human body is suspicious of undergoing the pre-drowning event, the respective human body may be further tracked, by the main control unit, in an additional subset of subsequent images corresponding to a predefined time interval (for example, 5-15 seconds, e.g., 10 seconds).);
and generating the alert is based on the level of the critical event
(Col 18 lines 15-25 Unless during the predefined time interval at least one of conditions (i)-(iv) described above with respect to FIG. 7A is unmet, the main control unit may determine that the human body is actually undergoing the pre-drowning event and may cause the alarm unit to issue an alarm.).
Claim 11. Ben teaches a method for monitoring an environment having a body of water using a water surveillance system, the method comprising:
providing, by two or more imaging modules, image data of an environment comprising a
body of water
(Col 9 lines 30-40 system 100 may include two or more camera units 110, wherein camera 112 of each of two or more camera units 110 may be directly connected to main control unit 120.);
receiving, by a processing unit communicatively connected to the two or more imaging
modules, the image data of the environment from each of the two or more imaging modules
(Col 11 lines 60-67 (58) The method may include receiving 204, by a main control unit, the plurality of images from the at least one camera (e.g., main control unit 120 as described above with respect to FIGS. 1A, 1B and 1C).);
identifying, by a neural network of the processing unit, one or more objects within the
environment based on the image data, wherein identifying the one or more objects comprises
associating an object identifier with each of the one or more objects
(Col 12 lines 20-25 The method may include detecting 208, by the main control unit, in an image of the plurality of images, one or more human bodies (e.g., as described below with respect to FIGS. 4A, 4B, 4C and 4D).
Col 14 lines 1-5, 40-50 pre-trained artificial intelligence (AI) body parts detection model. For example, the AI body parts detection model 420 may, for example, include a neural network 422,...The method may include defining 438, by the main control unit, two or more body part bounding boxes, each body part bounding box bounds one of the two or more detected body parts. (e.g. bounding boxes identifying human));
determining, by the processing unit, status data of the one or more objects based on the
image data
(Col 12 35-40 The method may include determining 212, by the main control unit, for each of the one or more human bodies, based on a first subset of images of the plurality of images, based on the tracking of the respective human body and based on a predefined set of rules, whether or not the respective human body is suspicious of undergoing a pre-drowning event or a drowning event.);
providing, by the processing unit, a distress parameter based on the object identifier
(Col 18 lines 1-40 (121) The human body may be suspicious of undergoing the pre-drowning event according to some embodiments of the invention if: i) the head (represented in FIG. 7A by head bounding box 712a) of the human body in images 710 is below virtual safety line 714, (ii) an orientation of the human body (represented in FIG. 7A by body bounding box 712) in images 710 is vertical (or substantially vertical), (iii) a measure of motion of the human body in a vertical direction (represented in FIG. 7A by an arrow 715a) in images 710 is greater than a measure of motion of the human body in a horizontal direction (represented in FIG. 7A by an arrow 715b) in images 710, and (iv) a total measure of motion of the human body in images 710 is below a predefined motion threshold.
e.g. I,ii,iii,iv are distressed parameters);
determining, by the processing unit, a critical event related to at least one of the objects of
the one or more objects based on the status data and a distress parameter
(Col 18 lines 15-60 Upon determination that the human body is suspicious of undergoing the pre-drowning event, the respective human body may be further tracked, by the main control unit... Upon determination that the human body is suspicious of undergoing the drowning event, the respective human body may be further tracked, by the main control unit, in an additional subset of subsequent images corresponding to a predefined time interval (for example, 5-15 seconds, e.g., 10 seconds). );
and generating, by the processing unit, an alert based on the determination of the critical
event
(Col 18 lines 15-65 the main control unit may determine that the human body is actually undergoing the pre-drowning event and may cause the alarm unit to issue an alarm....the main control unit may determine that the human body is actually undergoing the drowning event and may cause the alarm unit to issue an alarm.).
Claim 13. The method of claim 11, wherein the system further comprises a display having a user
interface providing graphics
(Col 21 lines 1-10 (148) Input devices 935 may be or may include a mouse, a keyboard, a touch screen or pad or any suitable input device. It will be recognized that any suitable number of input devices may be operatively connected to computing device 900 as shown by block 935. Output devices 940 may include one or more displays, speakers and/or any other suitable output devices.)
, and wherein: determining the status data of the one or more objects comprises tracking, using a tracking algorithm, a movement of the one or more objects within the environment
(Col 4 lines 5-10 based on the tracking of the respective human body and a predefined set of rules, whether or not the respective human body is suspicious of undergoing a pre-drowning event or a drowning event;
Col 9 lines 50-60 Main control unit 120 may determine, based on the received images and a predetermined set of rules, whether or not at least, one of the one or more human bodies undergoes a pre-drowning or a drowning event (e.g., as described below with respect to FIG. 2 and FIGS. 7A, 7B, 7C, 7D, 7E).)
determining the critical event further comprises comparing the movement of the one or more objects to a threshold of the distress parameter
(Col 15 lines 55-65 (i) the respective body bounding boxes in the respective two subsequent images have a maximal overlap with each other as compared to overlaps between other body bounding boxes in the images and (ii) an overlap between the respective body bounding boxes in the respective two subsequent images is above a specified threshold (e.g., as schematically shown in FIG. 5B). Step 510 may be done using, for example, intersection over union (IoU) method (e.g., as schematically shown in FIG. 5B). Setting the minimal specified threshold (e.g., 5-10%) );
the tracking changes a corresponding weight of each level of a cascade matching algorithm based on a current situation, a status of the system, and/or detection characteristics
(Col 16 lines 60-67 Col 17 lines 1-25 e.g. sequence of body parts are detected (cascade matching algorithm)
Col 18 lines 40-50 (i) the orientation of the human body (represented in FIG. 7B by bounding box 722) in images 710 is horizontal (or substantially horizontal); (ii) the human body (represented in FIG. 7B by hounding box 722) in images 710 is below virtual safety line 724; and (iii) a total measure of motion of the human body (represented in FIG. 7A by an arrow 725a) in images 720 is below a predefined motion threshold.);
and the method further comprises rendering, by the processing unit, on the display, a three-
dimensional (3D) tracking view on the display comprising visual indicators for each of the one
or more objects in the environment based on the tracking and/or object identifiers of the one or
more objects
Col 5 lines 25-40 define, a first body part bounding box that bounds the first body part and a second body part bounding box that bounds the second body part in the image; calculate a first centroid point of the first body part bounding box and a second centroid point of the second body part bounding box in the image; determine an angle between (i) a line extending between the first centroid point and the second centroid point, and (ii) a virtual horizontal line in the image; and determine, based on the determined angle, an orientation of the human body in the image.
Col 17 lines 60-65 FIG. 7A schematically shows a subset 710 of subsequent underwater images 710a, 710b, 710c including a detected human body represented by a body bounding box 712. (e.g. 3D image shown with object identifiers)).
Claim 14. Ben teaches the method of claim 11, wherein: the status data comprises a current location of the one or more objects and/or a submergence of the one or more objects
(Col 4 lines 1-5 a position and orientation of the respective human body with respect to the virtual safety line and a measure of motion of the respective human body);
the distress parameter comprises a predetermined duration of time for submersion
beneath a surface of the body of water
(Col 18 lines 15-25 in an additional subset of subsequent images corresponding to a predefined time interval (for example, 5-15 seconds, e.g., 10 seconds).);
And determining the critical event comprises determining that a first object of the one or more
objects has been submerged beneath the surface of the body of water for more than the
predetermined duration of time
(Col lines (i) the head (represented in FIG. 7A by head bounding box 712a) of the human body in images 710 is below virtual safety line 714,.. an additional subset of subsequent images corresponding to a predefined time interval (for example, 5-15 seconds, e.g., 10 seconds).).
Claim 17. Ben teaches the method of claim 11, wherein:
the critical event comprises an object of the one or more objects being underwater for too
long, approaching an edge of the body of water as a non-swimmer, running on wet pavement,
and/or jumping or diving on another object
(Col 17 lines 55-60 Reference is now made to FIG. 7A, which schematically shows a subset 710 of subsequent underwater images including a human body that is suspicious of undergoing a pre-drowning event, according to some embodiments of the invention.);
and determining the critical event comprises identifying a level of the critical event
(Col 18 lines 15-25 Upon determination that the human body is suspicious of undergoing the pre-drowning event, the respective human body may be further tracked, by the main control unit, in an additional subset of subsequent images corresponding to a predefined time interval (for example, 5-15 seconds, e.g., 10 seconds).);
and generating the alert is based on the level of the critical event
(Col 18 lines 15-25 Unless during the predefined time interval at least one of conditions (i)-(iv) described above with respect to FIG. 7A is unmet, the main control unit may determine that the human body is actually undergoing the pre-drowning event and may cause the alarm unit to issue an alarm.).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2, 10, 12 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ben Gigi in view of Tang (US 20160340006 A1).
Claim 2. Ben teaches the system of claim 1, and further discloses the use of multiple cameras to obtain certain angles and using an infrared cut-cut filter for at least one camera ([Col 9 lines 10-15]) but does not specifically disclose wherein the two or more imaging modules comprises at least:
an infrared imaging module, wherein the infrared imaging module is configured to
provide infrared image data;
a visible spectrum imaging module, wherein the visible spectrum imaging module is configured to provide visible spectrum image data;
and wherein each of the two or more imaging modules are positioned at a different location
relative to each other to provide a different angle of view of the environment.
However, Tang teaches an infrared imaging module, wherein the infrared imaging module is configured to provide infrared image data
([0131]multiple cameras including infrared and visible light cameras placed on ship structures that detect areas 1006);
a visible spectrum imaging module, wherein the visible spectrum imaging module is configured to provide visible spectrum image data
([0131]multiple cameras including infrared and visible light cameras placed on ship structures that detect areas 1006);
and wherein each of the two or more imaging modules are positioned at a different location
relative to each other to provide a different angle of view of the environment
([0137] Multiple pairs of visible light and infrared cameras are positioned strategically on the ship structure, pointing at the sea to monitor any overboard event and to provide the initial estimated position of the event.).
Therefore, it would have been obvious to one ordinarily skilled in the art before the effective filing date of invention to use visible, infrared images modules positioned at a different location
relative to each other to provide a different angle of view of the environment taught by Tang within the system of Ben for the purpose of utilizing double detection measures of an individual visibly in water and the heat signature generated by the individual to confirm the presence of the individual within the water.
Claim 10. Ben teaches the system of claim 1, and further discloses the capturing of images for display to a user but does not specifically disclose wherein:
a pixel density of the image data is large; and the two or more imaging modules are further configured to: in response to a zoom signal, generate a crop of the image data, wherein the crop:
(i) is a desirable framing of the environment, and (ii) associates a desirable amount of the
image data with each of the one or more objects; and wherein the processing unit is further configured to send the zoom signal in response to a distance between one of the two or more objects and one of the one or more objects meeting a distance threshold or a classification quality of the neural network meeting a precision threshold.
However, Tang teaches a pixel density of the image data is large
([0143] available LWIR camera with a 640×480 pixels image size... the maximum identification range (i.e. the range that still clearly identify the characteristics of the object) can be as large as 435 meters, which is sufficient to cover the water surrounding the entire ship.);
and the two or more imaging modules are further configured to:
in response to a zoom signal, generate a crop of the image data, wherein the crop:
(i) is a desirable framing of the environment, and (ii) associates a desirable amount of the
image data with each of the one or more objects
([0145] Note that the camera starts with a wide viewing angle to ensure the target is in the field of view and then zoom in to a relatively small view angle once the target is locked. (e.g. zoom and desirable framing) ..
The computers connected to these IR cameras continuously apply the tracking algorithms to the acquired images to look for the region with the highest correlation with the thermal features of human—this is called the “detector” mode. (associates desirable amount));
And wherein the processing unit is further configured to send the zoom signal in response to a
distance between one of the two or more objects and one of the one or more objects meeting a
distance threshold or a classification quality of the neural network meeting a precision threshold
([0115] The live audio/video stream 314 is transmitted wirelessly to the command center for image and acoustic processing in step 308.
[0144] If the correlation is beyond a certain threshold, the computers declare an alarm and switch to the “follower” mode. In the “follower” mode, the computer calculates the position of the target relative to the center of the frame and adjusts the cameras to re-lock the target in the center of the frame. ).
Therefore, it would have been obvious to one ordinarily skilled in the art before the effective filing date of invention to use pixels and the computing unit as taught by Tang within the system of Ben for the purpose of enhancing the system to use a wider field of view to detect for a specific object under duress and to specify the exact location of the object following the detection.
Claim 12. Ben teaches the method of claim 11, and further discloses the use of multiple cameras to obtain certain angles and using an infrared cut-cut filter for at least one camera ([Col 9 lines 10-15]) but does not specifically disclose wherein the two or more imaging modules comprises at least: an infrared imaging module, wherein the infrared imaging module is configured to provide infrared image data;
a visible spectrum imaging module, wherein the visible spectrum imaging module is configured to provide visible spectrum image data; and wherein each of the two or more imaging modules are positioned at a different location relative to each other to provide a different angle of view of the environment
However, Tang teaches wherein the two or more imaging modules comprises at least: an infrared imaging module, wherein the infrared imaging module is configured to
provide infrared image data
([0131]multiple cameras including infrared and visible light cameras placed on ship structures that detect areas 1006);
a visible spectrum imaging module, wherein the visible spectrum imaging module is
configured to provide visible spectrum image data
([0131]multiple cameras including infrared and visible light cameras placed on ship structures that detect areas 1006);
And wherein each of the two or more imaging modules are positioned at a different location
relative to each other to provide a different angle of view of the environment
([0137] Multiple pairs of visible light and infrared cameras are positioned strategically on the ship structure, pointing at the sea to monitor any overboard event and to provide the initial estimated position of the event.).
Therefore, it would have been obvious to one ordinarily skilled in the art before the effective filing date of invention to use visible, infrared images modules positioned at a different location
relative to each other to provide a different angle of view of the environment taught by Tang within the system of Ben for the purpose of utilizing double detection measures of an individual visibly in water and the heat signature generated by the individual to confirm the presence within the water.
Claim 20. Ben teaches the method of claim 11, and further discloses the capturing of images for display to a user but does not specifically disclose wherein: a pixel density of the image data is large; and
the two or more imaging modules are further configured to: in response to a zoom signal, generate a crop of the image data, wherein the crop: (i) is a desirable framing of the environment, and (ii) associates a desirable amount of the image data with each of the one or more objects; and
the method further comprises sending, by the processing unit, the zoom signal in response to a distance between one of the two or more objects and one of the one or more objects meeting a distance threshold or a classification quality of the neural network meeting a precision threshold.
However, Tang teaches a pixel density of the image data is large
([0143] available LWIR camera with a 640×480 pixels image size... the maximum identification range (i.e. the range that still clearly identify the characteristics of the object) can be as large as 435 meters, which is sufficient to cover the water surrounding the entire ship.);
; and the two or more imaging modules are further configured to:
in response to a zoom signal, generate a crop of the image data, wherein the crop:
(i) is a desirable framing of the environment, and (ii) associates a desirable amount of the
image data with each of the one or more objects
([0145] Note that the camera starts with a wide viewing angle to ensure the target is in the field of view and then zoom in to a relatively small view angle once the target is locked. (e.g. zoom and desirable framing) ..
The computers connected to these IR cameras continuously apply the tracking algorithms to the acquired images to look for the region with the highest correlation with the thermal features of human—this is called the “detector” mode. (associates desirable amount));
And the method further comprises sending, by the processing unit, the zoom signal in
response to a distance between one of the two or more objects and one of the one or more objects
meeting a distance threshold or a classification quality of the neural network meeting a precision
threshold
([0115] The live audio/video stream 314 is transmitted wirelessly to the command center for image and acoustic processing in step 308.
[0144] If the correlation is beyond a certain threshold, the computers declare an alarm and switch to the “follower” mode. In the “follower” mode, the computer calculates the position of the target relative to the center of the frame and adjusts the cameras to re-lock the target in the center of the frame. ).
Therefore, it would have been obvious to one ordinarily skilled in the art before the effective filing date of invention to use pixels and the computing unit as taught by Tang within the system of Ben for the purpose of enhancing the system to use a wider field of view to detect for a specific object under duress and to specify the exact location of the object following the detection.
Claim(s) 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Ben, Shlomovitz (US 20210327246 A1) and further in view of Brown (US 4233677 A).
Claim 5. Ben teaches the system of claim 1, further comprising an audio module having at least:
a speaker (Col 21 lines 5-10 speakers and/or any other suitable output devices.).
Ben further discloses other input/output devices but does not specifically disclose a microphone configured to provide audio data to the processing unit.
However, Shlomovitz teaches a microphone configured to provide audio data to the processing unit and wherein the audio module is communicatively connected to the processing unit
([0026] In embodiments the processing center may provide for implementing a proprietary signal processing method of the acoustic signals received from the hydrophone array and/or surface microphones in order to monitor, detect unauthorized entry into the body of water. In embodiments, the processing center may further provide for identifying the location of the unauthorized entry.
[0087] system 100 may further comprise at least one or more optional sensor(s) in addition to the hydrophone 101).
Therefore, it would have been obvious to one ordinarily skilled in the art before the effective filing date of invention to use a microphone as taught by Shlomovitz within the system of Ben for the purpose of enhancing the system to detect audio sounds and intrusion into the pool.
Ben and Shlomovitz teaches the use of microphone within water but does not specifically disclose the processing unit configured to: detect a status of a communicative connection of the audio module with the processing unit; and generate, if the status comprises a malfunction status, a verbal warning announcing a failure of the communicative connection; and wherein determining the critical event is further based on the audio data.
However, Brown teaches the processing unit configured to: detect a status of a communicative connection of the audio module with the processing unit; and generate, if the status comprises a malfunction status, a verbal warning announcing a failure of the communicative connection; and wherein determining the critical event is further based on the audio data.
(Col 2 lines 10-15 again the alarm network and printer can be activated in the manner previously described to warn the operator of a possible hydrophone section malfunction.
Col 9 lines 25-30 All the above cause an alarm (a series of "beeps") to sound and the malfunction diagnosis to be printed out in manner of the output of FIG. 5.)
Therefore, it would have been obvious to one ordinarily skilled in the art before the effective filing date of invention to use the process of generating status of a malfunction as taught by Brown within the system of Ben and Shlomovitz for the purpose of alerting a user to replace the microphone device.
Claim 15. Ben teaches the method of claim 11, wherein the system further comprises an audio module having at least:
a speaker (Col 21 lines 5-10 speakers and/or any other suitable output devices.).
Ben further discloses other input/output devices but does not specifically disclose;
a microphone configured to provide audio data to the processing unit.
However, Shlomovitz teaches a microphone configured to provide audio data to the processing unit and wherein the audio module is communicatively connected to the processing unit
([0026] In embodiments the processing center may provide for implementing a proprietary signal processing method of the acoustic signals received from the hydrophone array and/or surface microphones in order to monitor, detect unauthorized entry into the body of water. In embodiments, the processing center may further provide for identifying the location of the unauthorized entry.
[0087] system 100 may further comprise at least one or more optional sensor(s) in addition to the hydrophone 101).
Therefore, it would have been obvious to one ordinarily skilled in the art before the effective filing date of invention to use a microphone as taught by Shlomovitz within the system of Ben for the purpose of enhancing the system to detect audio sounds and intrusion into the pool.
Ben and Shlomovitz teaches the use of microphone within water but does not specifically disclose wherein the method further comprises: detecting a status of a communicative connection of the audio module with the processing unit; and generating, if the status comprises a malfunction status, a verbal warning announcing a failure of the communicative connection; and wherein determining the critical event is further based on the audio data.
However, Brown teaches the processing unit configured to: detect a status of a communicative connection of the audio module with the processing unit; and generate, if the status comprises a malfunction status, a verbal warning announcing a failure of the communicative connection; and wherein determining the critical event is further based on the audio data.
(Col 2 lines 10-15 again the alarm network and printer can be activated in the manner previously described to warn the operator of a possible hydrophone section malfunction.
Col 8 lines 45-55 In the passive condition, the hydrophone sections X.sub.1, X.sub.2, . . . X.sub.n of FIG. 1 are evaluated in accordance with five (5)test parameters previously indicated, and these test parameters for the most part are measured against absolute standards. If in the evaluation stages the parameters are not within preselected values, internal flags are set within the computer. Result: visual and audio alarms are activated. (e.g. malfunction status)
Col 9 lines 25-30 All the above cause an alarm (a series of "beeps") to sound and the malfunction diagnosis to be printed out in manner of the output of FIG. 5. (e.g. beeps as verbal))
Therefore, it would have been obvious to one ordinarily skilled in the art before the effective filing date of invention to use the process of generating status of a malfunction as taught by Brown within the system of Ben and Shlomovitz for the purpose of alerting a user to replace the microphone device.
Claim(s) 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Ben in view of Mor (US 20200053320 A1).
Claim 6. Ben teaches the system of claim 1, and further discloses the process of identifying a condition of a swimmer but does not specifically disclose a database and a feature extractor convolutional neural network, wherein: the identifying further comprises storing, in the database, identification information for each of the one or more objects, wherein the identification information is: (i) associated with the object identifier of one of the one or more objects and (ii) comprises an age of the one or more objects, a presence of a supervisor, and/or a swimming skill level of the one or more objects; and providing the distress parameter based on the object identifier is further based on the identification information associated with the object identifier in the database.
However, Mor teaches a database and a feature extractor convolutional neural network, wherein: the identifying further comprises storing, in the database, identification information for each of the one or more objects, wherein the identification information is: (i) associated with the object identifier of one of the one or more objects and (ii) comprises an age of the one or more objects, a presence of a supervisor, and/or a swimming skill level of the one or more objects
([0007] Optionally, the analysis may comprise extracting descriptor about the each swimmer from a database, wherein the descriptor may comprise at least one of the following: medical information and depiction of normal swimming activity; and wherein the risk level of each swimmer may be determined based on the descriptor of the each swimmer.
[0146] the information may be extracted from visual input obtained from Camera 320 and Camera 322.
[0147] The descriptor may comprise demographic information of the swimmers, such as gender, age group, height, or the like...Swimmers 331 and 332 may be determined to be in an age group of children, while Swimmers 333, 334 and 335 may be determined as adults. ).
; and providing the distress parameter based on the object identifier is further based on the identification information associated with the object identifier in the database
([0082] On Step 125, a risk level of each swimmer may be determined. In some exemplary embodiments, the risk level of each swimmer may be related to drowning potential of the each swimmer. The risk level of each swimmer may be determined based on the descriptor of the each swimmer, the presence of a spotter observing the swimmer, the field of view of the spotter, the location of the swimmer in the water area, or the like.
[0084] Such situations may be associated with higher severity level that cause the system to increase its sensitivity and give more focus to the event, in order to carefully monitor if the event is developing into an actual distress.
[0122] On Step 165, the location of the swimmer may be illuminated by the visual indicator. As a result, the lifeguard or the emergency personal may be enabled to see the alert without losing eye contact with the water area. The lifeguard or the emergency personal may be directed to the source of the distress, e.g., the location of the drowning event, and can reach the swimmer faster.)
Therefore, it would have been obvious to one ordinarily skilled in the art before the effective filing date of invention to use a database, a feature extractor and the identification information as taught by Mor within the system of Ben for the purpose of gathering all of the information related to a swimmer so that a supervisory person can be notified of an impending risk and event of the swimmer based on their physical attributes.
Claim 16. Ben teaches the method of claim 11, and further discloses the process of identifying a condition of a swimmer but does not specifically disclose wherein the system further comprises a database and a feature extractor convolutional neural network, and wherein:
the identifying further comprises storing, in the database, an identification information for
each of the one or more objects, wherein the identification information is: (i) associated with the
object identifier of one of the one or more objects and (ii) comprises an age of the one or more
objects, a presence of a supervisor, and/or a swimming skill level of the one or more objects; and
providing the distress parameter based on the object identifier is further based on the
identification information associated with the object identifier in the database.
However, Mor teaches a database and a feature extractor convolutional neural network, and wherein: the identifying further comprises storing, in the database, an identification information for
each of the one or more objects, wherein the identification information is: (i) associated with the
object identifier of one of the one or more objects and (ii) comprises an age of the one or more
objects, a presence of a supervisor, and/or a swimming skill level of the one or more objects;
([0007] Optionally, the analysis may comprise extracting descriptor about the each swimmer from a database, wherein the descriptor may comprise at least one of the following: medical information and depiction of normal swimming activity; and wherein the risk level of each swimmer may be determined based on the descriptor of the each swimmer.
[0146] the information may be extracted from visual input obtained from Camera 320 and Camera 322.
[0147] The descriptor may comprise demographic information of the swimmers, such as gender, age group, height, or the like...Swimmers 331 and 332 may be determined to be in an age group of children, while Swimmers 333, 334 and 335 may be determined as adults. ).
and providing the distress parameter based on the object identifier is further based on the
identification information associated with the object identifier in the database
([0082] On Step 125, a risk level of each swimmer may be determined. In some exemplary embodiments, the risk level of each swimmer may be related to drowning potential of the each swimmer. The risk level of each swimmer may be determined based on the descriptor of the each swimmer, the presence of a spotter observing the swimmer, the field of view of the spotter, the location of the swimmer in the water area, or the like.
[0084] Such situations may be associated with higher severity level that cause the system to increase its sensitivity and give more focus to the event, in order to carefully monitor if the event is developing into an actual distress.
[0122] On Step 165, the location of the swimmer may be illuminated by the visual indicator. As a result, the lifeguard or the emergency personal may be enabled to see the alert without losing eye contact with the water area. The lifeguard or the emergency personal may be directed to the source of the distress, e.g., the location of the drowning event, and can reach the swimmer faster.)
Therefore, it would have been obvious to one ordinarily skilled in the art before the effective filing date of invention to use a database, a feature extractor and the identification information as taught by Mor within the system of Ben for the purpose of gathering all of the information related to a swimmer so that a supervisory person can be notified of an impending risk and event of the swimmer based on their physical attributes.
Allowable Subject Matter
Claims 8,9,18 and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
For claim 8, Mor (US 20200053320 A1) was the closest prior art to teach the process of detecting obstructions and detecting gestures of swimmers needing assistance and using neural network and perform an analysis to discover events. However, the prior art fails to specifically teach all of the combined features of claim 8, wherein the processing unit is further configured to:
determine an updated critical event based on an updated status data, wherein the updated
critical event comprises a deescalated critical event, wherein the deescalated critical event
comprises an object of the one or more objects re-surfacing above water before a specific
predetermined duration of time for submergence is exceeded, a command gesture of a user, a
presence of a supervisor, and/or a command gesture of the supervisor;
identify an obstruction blocking at least a portion of a field of view (FOV) of the two or
more imaging modules;
generate, in response to the identifying the obstruction, a notification to instruct a user to
remove the obstruction from the at least a portion of the field of view of the one or more imaging
modules; and
report a functionality status to a communicatively connected cloud server to enable the
cloud server to alert a user if a malfunction of the system is detected; and
wherein identifying each object of the one or more objects within the environment further
comprises:
re-identifying the object if the object has been previously identified; and
re-identifying the object as the same object if the object has been occluded by the
obstruction and re-appeared in a field of view of one or more of the imaging modules;
and wherein the neural network comprises a feature extractor neural network.
For Claim 9, Ben was the closest prior art to teach the process of classifying the object; determining a submergence level of the object and a center contact point of the object if the submergence level exceeds a depth threshold; the head and robustness of the object. However, the prior art fails to specifically teach the command gestures, the user input and the mode of operation comprises gamification, communication, entertainment modes, a non-pool-time setting, a pool-time setting, an out-of-season setting, and a good-swimmers-only setting, and wherein the gamification mode comprises
Simon says, red-light-green-light, and race coordination games.
Mor was the closest prior art to teach the process of detecting obstructions and detecting gestures of swimmers needing assistance and using neural network and perform an analysis to discover events. However, the prior art fails to specifically teach wherein: the identifying each object of the one or more objects comprises: (i) classifying the object as a user, supervisor, other person, or animal, (ii) determining a submergence level of the object and a center contact point of the object if the submergence level exceeds a depth threshold; (iii) a head of the object, and/or (iv) a robustness of the identifying of the object; the status data comprises an age of each of the one or more objects, a context of each of a presence of the one or more objects, a direct interaction of each of the one or more objects with one of the one or more objects, a movement style of each of the one or more objects, a sound
made of each of the one or more objects, a directive given by the supervisor, a user-selected
mode of operation, a swimming skill level of each of the one or more objects, an identification of
each of the one or more objects, a location of each of the one of more objects within the
environment, a direction of each of the one or more objects, a speed of each of the one or more
objects, a medical condition risk, and/or a dangerous situation identification;
the processing unit is further configured to: determine that the image data comprises a visual representation of a command gesture from a user or the supervisor; identify the command gesture;
alter the status data of at least one of the one or more objects and/or the distress parameter based on the command gesture; and receive a user input comprising instructions to alter a mode of operation of the system, wherein the mode of operation comprises gamification, communication, entertainment modes, a non-pool-time setting, a pool-time setting, an out-of-season setting, and a good-swimmers-only setting, and wherein the gamification mode comprises Simon says, red-light-green-light, and race coordination games.
For Claim 18, the reason for objection and allowable subject matter is similar to that of claim 8.
For Claim 19, the reason for objection and allowable subject matter is similar to that of claim 9.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RUFUS C POINT whose telephone number is (571)270-7510. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Davetta Goins can be reached at 571-272-2957. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RUFUS C POINT/Primary Examiner, Art Unit 2689
Prosecution Insights