DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed on 01/12/2026 have been fully considered but they are not persuasive.
Applicant argues: “Independent Claim 21 recites, among other things, "displaying [an] image in a customer interface, [the image having] overlaid thereon a plurality of pre-sized boxes" … Critically, according to Yakupov, a bounding box is displayed based on contours that are "more closely matching the expected size of [a detected] object".4 This is clearly illustrated in Yakupov Figure 7E, wherein each of the six object detection frames (five people and one motorcycle) have different sizes. Thus, Yakupov cannot be understood as disclosing displaying an image having overlaid thereon a plurality of pre-sized boxes since the size of an object detection frame depends on the contour of the corresponding detected object.”
Examiner notes that Applicant’s argument presumes that the claimed pre-sized boxes are generated by a method other than the computer vision methods of the prior art. Examiner notes that this claim construction is unreasonable, because the claims provide no limitations on how to generate pre-sized boxes. Just as claimed, prior art pre-sizes the boxes for the user. See reasons for rejection below.
Applicant argues: “The Examiner further asserts that Yakupov ,m [0090] and [0091] disclose "receiving input ... specifying one or more of the plurality of pre-sized boxes, wherein the specified one or more pre-sized boxes define a zone".5 The Applicant respectfully traverses this assertion. Yakupov discloses that "the system can be configured to allow the user to restrict or exclude detection objects identified by the object detection engine".”
Examiner again notes that Applicant presumes “receiving input ... specifying one or more of the plurality of pre-sized boxes” is a particular method which is different from the selection performed by the prior art. However, as noted in the reasons for rejection below, the claims are not limited to a particular method of receiving input or specifying.
A claim can not be allowed based on details that are not claimed.
Applicant argues: “Yakupov further discloses that the system can allow a user to draw a zone for inclusion or exclusion using the dashboard display, for example as described with respect to Figure 5E, by using draw tool 220 to draw bounding box 219 that defines inclusion zone 218.10 Notably, the user-drawn bounding box 219 illustrated in Yakupov Figure 5E is not the same as the object detection frames illustrated in Yakupov Figures 7 A-7E, which the Examiner previously asserted as corresponding to the claimed "plurality of presized boxes".”
Examiner notes that in Yakupov, a user can draw a zone that selects some of the pre-sized boxes, which is within the scope of the claimed input “specifying one or more of the plurality of pre-sized boxes.” See reasons for rejection below.
Regarding the newly amended claim language: “The Applicant has amended independent Claim 35 to recite, among other things, a system comprising at least one first computing device that is configured to, among other things, "receive user input ... specifying one or more of [a] plurality of user-selectable pre-sized boxes [that] define a zone" (emphasis added).”
Examiner notes that Applicant has amended Claim 35 to be more consistent with Claim 21. The updated reasons for rejection below reflect this.
Applicant argues: “New Claim 42 depends from Claim 41 and further specifies that "communicating the representation of the zone to the second interface comprises communicating the data structure to the second interface". These new claims are added in response to the Examiner's suggestion, provided during the telephone interview, to claim "particular data structures for the boxes and the representations of the zone that contains those boxes"”
Examiner notes that the new claims do not limit “particular data structures for the boxes,” which Applicant argues to be the inventive aspect of the claim. The feature “data structure that includes one or more fields allocated to hold data identifying pixels included in the zone” is much broader and reads on an embodiment found in the prior art. See reasons for rejection of the new claims below.
Examiner suggests clarifying the claim limitations with respect to the data structures of the pre-sized boxes as they apply to the user interface of the independent claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 21-37, 40-42 are rejected under 35 U.S.C. 103 as being unpatentable over US 20200117905 to Yakupov in view of US 7366359 to Davey (“Davey”).
Regarding Claim 21: “A method comprising:
receiving an image of a field of view of an image capture device; (“computer is configured to interface with one or more cameras over the network, wherein the one or more cameras provide the one or more video streams.” Yakupov, Paragraph 12.)
displaying the image in a customer interface, wherein the image displayed in the customer interface has overlaid thereon a plurality of pre-sized boxes; (This claim does not limit the available number and dimensions of pre-sized boxes. Yakupov teaches “The scene detection engine is then configured to perform number 1 image manipulations in order to find the contours … Next, the device is configured to select contours more closely matching the expected size of the object … # compute the bounding box for the contour so it can be displayed,” thus the scene detection engine pre-sizes the bounding boxes to match the contours of the detected objects. See Yakupov, Paragraphs 79-80, 82 and the plurality of bounding boxes in Figs. 7-8.)
receiving input, via the customer interface, specifying one or more of the plurality of pre-sized boxes, wherein the specified one or more pre-sized boxes define a zone; and (This claim does not limit the method of specifying. Yakupov teaches “For example, in an embodiment, the system can be configured to allow the user to restrict or exclude [specify or select] detection objects identified [as a pre-sized bounding box] by the object detection engine 124 for each camera stream.” Yakupov, Paragraph 90. One way to perform that selection is: “to allow the user to draw a zone for inclusion or exclusion [selection] using the dashboard display, for example as described with respect to FIG. 5E, by using a draw tool 220 to draw a bounding box 219 that defines an inclusion zone 218” and any bounding boxes that are within the inclusion zone are selected. Yakupov, Paragraphs 91.)
communicating, to a monitor interface implemented by a computing device within a monitoring center, (“Computers can optionally communicate directly with another computer.” Yakupov, Paragraphs 113, 118. The monitor interface can be “A web-enabled computer can include a browser application that is configured to receive and to send web pages, web-based messages, and the like. The browser application can be configured to receive and display graphics, text, multimedia, and the like … a user of the client computer can employ the browser application to perform various activities over a network” Yakupov, Paragraph 112.)
a representation of the zone overlaid upon the image, (The claim does not provide a definition or limitation on the representation of the zone. Prior art performs an embodiment: “The system can be configured to ignore objects outside the zone, and only perform frame analysis and object detection inside the bounding box 219.” Yakupov, Paragraph 91 and Figs. 7-8. See additional treatment below.)
thereby providing customer privacy during event processing by preventing monitoring personnel from seeing portions of the image overlaid by the representation of the zone.” (“The system could then offer the user the opportunity to exclude the zone, for example, via the rule interface 207 shown in FIGS. 5A-E,” thereby preventing outside computers or personnel from seeing the excluded objects. Yakupov, Paragraphs 90, 92, 60 and Fig. 7A. See additional treatment below.)
Yakupov does not explicitly combine the above two functions, that the zone exclusion is implemented by overlaying a representation of the zone over the image displayed to a user, because its examples primarily target automated image processing. However, it is known in the art to mask (overlay) private image portions in order to prevent human operators from viewing sensitive information:
Davey teaches examples of that in the context of video security systems: “allowing an operator or other means to select a region for processing (preferably for privacy obfuscation, but potentially for other processing … and displaying the image or video with the privacy regions obscured [with a representation of the zone] … the privacy region is merged with the rest of the image or video” where a privacy region B’ is an obscured representation of the selected region in image A*’. Davey, Column 2, lines 6-16, Column 5, lines 21-41 and Figs. 7-8.
Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to supplement the teachings of Yakupov to perform image zone exclusion by overlaying a representation image portion over the real image portion, as taught in Davey, in order to mask off regions that should not be viewed in subsequent processes. See Davey, Column 1, lines 19-35 and Yakupov, Paragraphs 90, 92.
Finally, in reviewing the present application, there does not seem to be objective evidence that the claim limitations are particularly directed to: addressing a particular problem which was recognized but unsolved in the art, producing unexpected results at the level of the ordinary skill in the art, or any other objective indicators of non-obviousness.
Regarding Claim 22: “The method of Claim 21 , further comprising communicating, to the monitor interface, a portion of the image outside the zone, wherein a portion of the image inside the zone is not transmitted to the monitor interface.” (“The system could then offer the user the opportunity to exclude the zone, for example, via the rule interface 207 shown in FIGS. 5A-E,” thereby preventing outside computers or personnel from seeing the excluded objects. Yakupov, Paragraphs 90, 92, 60 and Fig. 7A. Also note “displaying the image or video with the privacy regions obscured [with a representation of the zone] … the privacy region is merged with the rest of the image or video” indicating that an obscuring representation and not the privacy image is transmitted to display. See Davey, Column 2, lines 6-16, Column 5, lines 21-41 and Figs. 7-8. See statement of motivation in Claim 21.)
Regarding Claim 23: “The method of Claim 21 , wherein the representation of the zone is communicated to a plurality of monitor interfaces implemented by one or more computing devices within the monitoring center.” (Note that this information can be communicated to multiple computer systems: “computers 101, 102 and 104-106 are not constrained to these services and can also be employed, for example, as an end-user computing node, in other embodiments. It should be recognized that more or less computers can be included within a system such as described herein, and embodiments are therefore not constrained by the number or type of client computers employed.” Here, “The computer further includes an input device such as, e.g., a keyboard or mouse, an output device such as, e.g., a CRT, LCD or LED display, a communications interface” Yakupov, Paragraphs 110, 107, and Fig. 1. See statement of motivation in Claim 1.)
Regarding Claim 24: “The method of Claim 21 , further comprising, before receiving the input, identifying a semantic region in the image, wherein the semantic region includes a set of pixels that depict one or more objects in the image.” (“The scene detection engine is then configured to perform number 1 image manipulations in order to find the contours [semantic regions] … Next, the device is configured to select contours more closely matching the expected size of the object … # compute the bounding box for the contour so it can be displayed,” thus the scene detection engine pre-sizes the bounding boxes to match the contours of the detected objects. See Yakupov, Paragraphs 79-80, 82 and the plurality of bounding boxes in Figs. 7-8.)
Regarding Claim 25: “The method of Claim 24, further comprising identifying the one or more objects as being immobile using semantic segmentation.” (“For example, the system can detect that a large portion [segment] of a frame is, for example, a wooded area or a highway outside a parking lot [semantic] being monitored. The system could then offer the user the opportunity to exclude the zone …” Yakupov, Paragraph 92.)
Regarding Claim 26: “The method of Claim 21 , further comprising, before receiving the input, identifying a semantic region in the image and associating the semantic region in the image with an object class.” (“when a new object appears to the camera 102 … the new object is classified (e.g. class: "person") with a detailed analysis 118.” Note that this occurs before the user is provided with object identifications and can draw the ignore zones. Yakupov, Paragraphs 77, 90.)
Regarding Claim 27: “The method of Claim 21 , further comprising receiving input, via the customer interface, requesting that the zone be stored as a customer zone by the computing device within the monitoring center.” (“the device is operatively connected to a rules engine and comprises an interface configured to allow a user to define an inclusion zone and an exclusion zone for object detection.” Yakupov, Paragraphs 19, 92.)
Claim 28, “A method,” is rejected for reasons stated for Claims 21 and 26.
Regarding Claim 29: “The method of Claim 28, further comprising:
receiving an additional image of the field of view of the image capture device; and (“the scene detection engine 120 is configured to perform object tracking by identifying the difference between a previous frame (nothing in it) and a current frame,” which is the additional frame in this case. “The system is then configured to repeat the process above for each new frame, until the frame sequences (scenes) stop changing.” Yakupov, Paragraphs 78, 81.)
redacting a portion of the additional image defined by the customer zone, thereby resulting in a redacted image.” (“displaying the image or video with the privacy regions obscured [redacted] … the privacy region is merged with the rest of the image or video” indicating that an obscuring representation and not the privacy image is transmitted to display. See Davey, Column 2, lines 6-16, Column 5, lines 21-41 and Figs. 7-8. “the corresponding pixels in the distorted image obscured for each subsequent frame.” Davey, Column 6, lines 1-3. See statement of motivation in Claim 21.)
Regarding Claim 30: “The method of Claim 29, further comprising displaying the redacted image in the monitor interface, thereby providing customer privacy during event processing by preventing monitoring personnel from seeing the portion of the additional image overlaid by the customer zone.” (“and displaying the image or video with the privacy regions obscured [redacted] … the privacy region is merged with the rest of the image or video” where a privacy region B’ is an obscured representation of the selected region in image A*’. Davey, Column 2, lines 6-16, Column 5, lines 21-41 and Figs. 7-8. See statement of motivation in Claim 21.)
Claim 31 is rejected for reasons stated for Claims 21 and 23 in view of the Claim 29 rejection.
Claims 32-34 are rejected for reasons stated for Claims 24-26 respectively in view of the Claim 29 rejection.
Claim 35, “A system comprising at least one first computing device” is rejected for reasons stated for Claim 21, because the “user input specifying one or more of the plurality of user-selectable pre-sized boxes” corresponds to “input via customer interface specifying one or more of the plurality of pre-sized boxes” of Claim 21, and because prior art teaches: “computers 101, 102 and 104-106 are not constrained to these services and can also be employed, for example, as an end-user computing node, in other embodiments. It should be recognized that more or less computers can be included within a system such as described herein, and embodiments are therefore not constrained by the number or type of client computers employed.” Yakupov, Paragraphs 110, 107, and Fig. 1.)
Regarding Claim 36: “The system of Claim 35, wherein:
the first interface is a customer interface; (Note that the word customer does not particularly limit the structure or function of the interface. Prior art teaches: “In one embodiment, a user of the client [customer] computer can employ the browser application to perform various activities over a network.” Yakupov, Paragraph 112.)
the at least one first computing device includes a mobile computing device; (“computing device … such as, laptop computer, smart mobile telephone, and tablet computers, and the like.” Yakupov, Paragraph 111.)
the second interface is a monitor interface; and … the at least one second computing device is implemented within a monitoring center.” (Note that the name of the computer location indicates a preferred use but does not limit the structure of the system. Cumulatively note that “FIG. 4A illustrates a graphical user [monitoring] interface with five camera feeds … As shown in FIG. 4A feed 202a is a Street feed, 202b is a Parking feed, and cameras 202c-202e are feeds from cameras being provided from servers in remote locations.” Yakupov, Paragraph 124. Thus, the monitoring computer location can be different and remote from the camera location or location of other computers and servers.)
Regarding Claim 37: “The system of Claim 36, wherein
the at least one first computing device is further configured to communicate, to the at least one second computing device, (As noted above, “a user of the client [customer] computer can employ the browser application to perform various activities over a network.” Yakupov, Paragraph 112. Note that this information can be communicated to multiple computer systems: “computers 101, 102 and 104-106 are not constrained to these services and can also be employed, for example, as an end-user computing node, in other embodiments. It should be recognized that more or less computers can be included within a system such as described herein, and embodiments are therefore not constrained by the number or type of client computers employed.” Yakupov, Paragrpah 110. See statement of motivation in Claim 21.)
a request to store the zone as a customer zone.” (“the device is operatively connected to a rules engine and comprises an interface configured to allow a user to define an inclusion zone and an exclusion zone for object detection.” Yakupov, Paragraphs 19, 92.)
Claim 38: “The system of Claim 35, wherein:
the first interface is a monitor interface; … the at least one first computing device is implemented within a monitoring center; (Note that the name of the computer location indicates a preferred use but does not limit the structure of the system. Cumulatively note that “FIG. 4A illustrates a graphical user [monitoring] interface with five camera feeds … As shown in FIG. 4A feed 202a is a Street feed, 202b is a Parking feed, and cameras 202c-202e are feeds from cameras being provided from servers in remote locations.” Yakupov, Paragraph 124. Thus, the monitoring computer location can be different and remote from the camera location or location of other computers and servers.)
the second interface is a customer interface; and (Note that the word customer does not particularly limit the structure or function of the interface. Prior art teaches: “In one embodiment, a user of the client [customer] computer can employ the browser application to perform various activities over a network.” Yakupov, Paragraph 112.)
the at least one second computing device includes a mobile computing device.” (“computing device … such as, laptop computer, smart mobile telephone, and tablet computers, and the like.” Yakupov, Paragraph 111.)
Regarding Claim 39: “The system of Claim 38, wherein
the at least one second computing device is further configured to communicate, to the at least one first computing device, (As noted above, “a user of the client [customer] computer can employ the browser application to perform various activities over a network.” Yakupov, Paragraph 112. Note that this information can be communicated to multiple computer systems: “computers 101, 102 and 104-106 are not constrained to these services and can also be employed, for example, as an end-user computing node, in other embodiments. It should be recognized that more or less computers can be included within a system such as described herein, and embodiments are therefore not constrained by the number or type of client computers employed.” Yakupov, Paragraph 110. See statement of motivation in Claim 21.)
a request to store the zone as a customer zone.” (“the device is operatively connected to a rules engine and comprises an interface configured to allow a user to define an inclusion zone and an exclusion zone for object detection.” Yakupov, Paragraphs 19, 92.)
Regarding Claim 40: “The system of Claim 35, further comprising
at least one third computing device that is implemented in a data center environment and that is configured to: (Note that the preferred named location of the computing device does not change the functionality of the system. Cumulatively, prior art teaches: “computers 101, 102 and 104-106 are not constrained to these services and can also be employed, for example, as an end-user computing node, in other embodiments. It should be recognized that more or less computers can be included within a system such as described herein, and embodiments are therefore not constrained by the number or type of client computers employed.” “remote computers and other related electronic devices could be remotely connected to either LANs or WANs via a modem and temporary telephone link. In one embodiment, network 110 can be configured to transport information of an Internet Protocol (IP).” Yakupov, Paragraphs 110, 119. See statement of motivation in Claim 21.)
receive a first subsequent image illustrative of the field of view of the image capture device; (“One process includes computing a mean structural similarity index between two images, for example a current video frame against a previous video frame,” Yakupov, Paragraph 71.)
identify a particular object in the first subsequent image; (“For example, in an embodiment, when a new object appears to the camera 102 (e.g. mailman), the system can be configured to track the new object after the new object is classified (e.g. class: "person") … invoke object tracking using the scene detection engine 120 analysis 114, which will track the "blob" as it moves across the screen, without passing the stream for object detection 118” for the subsequent images. Yakupov, Paragraph 77.)
receive a second subsequent image illustrative of the field of view of the image capture device; (“One process includes computing a mean structural similarity index between two images, for example a current video frame against a previous video frame,” Yakupov, Paragraph 71. “The system is then configured to repeat the process above for each new frame” Yakupov, Paragraph 81.)
identify the particular object in the second subsequent image; and (“For example, in an embodiment, when a new object appears to the camera 102 (e.g. mailman), the system can be configured to track the new object after the new object is classified (e.g. class: "person") … invoke object tracking using the scene detection engine 120 analysis 114, which will track the "blob" as it moves across the screen, without passing the stream for object detection 118” for the subsequent images. Yakupov, Paragraph 77. “The system is then configured to repeat the process above for each new frame” Yakupov, Paragraph 81.)
calculate a metric that indicates an amount of change between the first subsequent image and the second subsequent image with reference to at least one of the zone or a reference object appearing in both the first subsequent image and the second subsequent image.” (For example, “Tills camera 102a can also be tuned to set a scene threshold to ignore events triggered smaller scene differentials, for example as would be triggered by small animals,” exemplifying objects appearing in each frame. Or in another example, “set a color space analysis or scene threshold that ignores movements that are above a threshold that correlates to fast movement by larger objects.” Yakupov, Paragraph 84.)
Regarding Claim 41: “The system of Claim 35, wherein
the at least one first computing device is further configured (For example a server camera feed as in Yakupov, Paragraph 131 and Figs. 6A-6B.)
to record the zone as a data structure that includes one or more fields allocated to hold data identifying pixels included in the zone.” (“As shown in FIG. 7 A, upon selecting a frame from the report, the display shows the selected frame stored from the feed and a tag of the object detected [data identifying pixels] in the object detection frame. For example, FIG. 7A shows [pixels of] a single airplane was identified in the frame and the object detection frame labeled as an "airplane" by the object detection engine 124,” indicating a recorded data structure with fields for the pixels of the video frame and for the label of the object, which is “data identifying pixels included in the zone” as specified by the user in the claims above. See Yakupov, Paragraphs 131-132 and frame selection in Paragraph 91.)
Regarding Claim 42: “The system of Claim 41, wherein communicating the representation of the zone to the second interface includes communicating the data structure to the second interface.” (For example, the second interface can be the display, “the display shows the selected frame stored from the feed and a tag of the object detected in the object detection frame. For example, FIG. 7A shows a single airplane was identified in” in Yakupov, Paragraph 132 and Fig. 7A.)
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MIKHAIL ITSKOVICH whose telephone number is (571)270-7940. The examiner can normally be reached Mon. - Thu. 9am - 8pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Joseph Ustaris can be reached at (571)272-7383. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MIKHAIL ITSKOVICH/Primary Examiner, Art Unit 2483
Prosecution Insights