DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see Remarks page 12, filed 03/13/2026, with respect to the rejections of claims 1-20 under 35 U.S.C. 101 have been fully considered and are persuasive. The rejections of claims 1-20 under 35 U.S.C. 101 have been withdrawn.
Applicant's arguments, see Remarks pages 9-11, filed 03/13/2026, with respect to the rejections of amended claims 1 and 19-20 under 35 U.S.C. 103 have been fully considered but they are not persuasive.
On page 11 of Remarks, Applicant argues:
PNG
media_image1.png
387
764
media_image1.png
Greyscale
Examiner respectfully disagrees.
Paragraph 0114 of Knighton discloses “Although not illustrated, embodiments automatically advance through the various steps recorded in connection with user interfaces 400 a - 400 d until the recipe is completed, thereby guiding a chef though the ingredients, timing, and temperature characteristics of the recorded recipe. Also, although not shown, interfaces 400 g and 400 h could include additional elements from user interfaces 400 a - 400 d , such as temperature pins, an ingredient panel, an instruction panel, and the like. Also, although not illustrated, a timer may indicate how long a chef is to perform a current action or how long to pause for a current action,” Wherein the timer included in the user interface indicating the amount of time required for the current recipe action constitutes an identification of a numerical length of time for the recipe step.
In addition, Figure 4G, and similarly Figure 4H: step 417c, of Knighton displays the user interface on step 417a, with an indication of a next recipe step 417b and a timer with an estimated amount of time of 0:05 until the step 417a is completed and 417b starts. Since the next step timer indicates the amount time of until the next step begins, the next step timer indicates an amount of time between the current step 417a and the next step 417b. Thus, constituting an identification of a numerical length of time between each step.
Thus, Knighton discloses the limitations “providing for display, on a display of a computing device, a user-interface element displaying…(3) an identification of a numerical length of time of each of the plurality of steps and (4) an identification of a numerical length of time between each step”.
Therefore the rejection of claim 1, under 35 U.S.C. 103 is maintained.
As per claim(s) 19 & 20, arguments made in rejecting claim(s) 1 are analogous.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 5-7, 10, and 12-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Knighton et al. (US2021251263A1) hereinafter referenced as Knighton, in view of Voigtlaender et al. (MOTS: Multi-Object Tracking and Segmentation) hereinafter referenced as Voigtlaender.
Regarding claim 1, Knighton discloses: A method comprising: capturing, by a camera, an image of a cooking surface, wherein the image is associated with a field of view that includes the cooking surface (Knighton: 0035: “When the sensory array 209 is mounted overhead, the thermal sensor(s) 210 and/or visible light sensor(s) 211 can be appropriately zoomed as needed to get the food preparation surface in the view.”; 0042: “one or more of the sensory data collection components 207 implement a visible light image capture module capable of capturing images/video from the visible light sensor(s) 211.”; Wherein both visible light and thermal sensors capture images of the surface);
identifying, based at least on the image, one or more cookware items within the field of view (Knighton: 0043: “one or more of the sensory data analysis components 208 use sensor data collected from the thermal sensor(s) 210 and from the visible light sensor(s) 211 to generally detect movement or changes to an observed object, such as the addition of cookware”);
associating, in real time, a unique identifier with each identified cookware item in the image associated with the field of view, wherein each unique identifier distinguishes its associated cookware item from every other cookware item in the image (Knighton: Figure 3; 0094-0095: “act 302 is performed by the UI computing device 202 or the cooking assistance service 203 (i.e., based on the accessory device 201 having sent at least one of the determined temperature, identity, or physical property of the food preparation surface or the object to at least one of the UI computing device 202 or the cooking assistance service 203). The physical property of the object can comprise any property detectible visually, such as at least one of a color of the object, a size of the object, or a thickness of the object.”; Wherein the method detecting the properties of the object while the user is cooking constitutes real-time detection, and wherein the identified object properties are unique);
capturing, by the camera, one or more subsequent image of the cooking surface; tracking, using the one or more subsequent images of the cooking surface, a location of each of the one or more cookware items relative to the cooking surface (Knighton: 0058: “one or more of the cooking assistant components 212 implement object detection (e.g., in connection with one or more of the sensory data analysis components 208 and/or one or more of the ML engines 205) …object detection also includes the ability to track objects on a per burner basis.”; Wherein objects detected are tracked by subsequent images);
determining, from at least one of the one or more subsequent images of the cooking surface, an identification of one or more subsequent cookware items in the at least one subsequent image (Knighton: 0042: “one or more of the sensory data collection components 207 implement a visible light image capture module capable of capturing images/video from the visible light sensor(s) 211. In embodiments, one or more of the sensory data analysis components 208 implement a visible light image processing module capable of processing this visual data, such as by feeding to an object detection algorithm (e.g., using one or more of the ML engines 205).”);
identifying each cookware item in the accessed image as one of (1) a removed cookware item (Knighton: 0059: “the embodiments may detect a pan, food items, and even seasoning. Yet another example of object detection includes detecting when pans are removed from the cooking burner or even detecting the cooking burner type (gas, electric, induction).”; Wherein the detection of a pan being removed from the burner constitutes the identification of a removed cookware item.) or (2) a cookware item in the at least one subsequent image (Knighton: 0107: “user interface 400a also illustrates a temperature pin 410 showing a point temperature of 304° F. for a single point in the cookware 402, along with a duration (30 seconds) for which the temperature pin 410 has been active. In embodiments, user interface 400 a enables manual and/or automatic placement of any number of temperature pins, and these temperature pins automatically move to track the object to which they are associated.”; Wherein the previously detected cookware is subsequently detected and tracked in subsequent images.);
identifying each cookware item in the at least one subsequent image as one of (1) a newly introduced cookware item (Knighton: 0043: “one or more of the sensory data analysis components 208 use sensor data collected from the thermal sensor(s) 210 and from the visible light sensor(s) 211 to generally detect movement or changes to an observed object, such as the addition of cookware, the addition of an ingredient, the flipping of an ingredient, the stirring of an ingredient, etc.”; Wherein the addition of cookware constitutes newly introduced cookware) or (2) a cookware item in the accessed image (Knighton: 0107: “user interface 400 a enables manual and/or automatic placement of any number of temperature pins, and these temperature pins automatically move to track the object to which they are associated.”; Wherein the tracking of an object constitutes the identification of previously detected cookware.);
providing for display, on a display of a computing device, a user-interface element displaying (1) an identification of a recipe (2) an identification of each of a plurality of steps of the recipe (Knighton: 0111: “FIGS. 4E and 4F illustrate example user interfaces 400 e 400 f that may be presented as part of recipe selection. Referring to FIG. 4E, illustrated is a user interface 400 e that includes a selection of available recipes 415 a - 415 c…FIG. 4F illustrates a user interface 400 f that may be displayed after selection of recipe 415 b , including a recipe information panel 416 that presents information, such as necessary ingredients, time to cook, a number of calories (e.g., as determined by the ingredients), ratings and/or reviews (e.g., as determined by social media features), and an overview of the recipe preparation process.”;
0112: “FIGS. 4G and 4H illustrate example user interfaces 400 g and 400 h that may be presented as part of a “scripted” AR recipe instruction session…User interface 400 g also includes an overlay of recipe steps 417 and instruction video section 418.”) (3) an identification of a numerical length of time of each of the plurality of steps (Knighton: 0114: “a timer may indicate how long a chef is to perform a current action or how long to pause for a current action.”) and (4) an identification of a numerical length of time between each step (Knighton: Figures 4G & 4H: 417b & 417c:
PNG
media_image2.png
462
638
media_image2.png
Greyscale
;
0112: “ In embodiments, a visual size of each recipe step 417 indicates which recipe step 417 is current (e.g., with the current recipe step 417 being visually larger than others), or an estimated relative duration of each recipe step 417.”
Wherein during a cooking step, the user interface contains a display indicating the estimated amount of time between the current step and the subsequent step.); and
updating, based on at least one of the identified cookware items, the user-interface element
with feedback regarding a cooking process involving the cooking surface and the one or more cookware items on a display of a computing device (Knighton: Figure 4A; 0107: “As an AR feature, user interface 400a also illustrates a temperature pin 410 showing a point temperature of 304° F. for a single point in the cookware 402…user interface 400a enables manual and/or automatic placement of any number of temperature pins, and these temperature pins automatically move to track the object to which they are associated.”).
Knighton does not disclose expressly: associating, in real time, a unique identifier with each identified cookware item in the at least one subsequent image, wherein each unique identifier distinguishes its associated cookware item from every other cookware item in the subsequent image; and identifying each cookware item in the accessed image as one of (1) a removed cookware item or (2) a cookware item in the at least one subsequent image, based on (1) a particular match, between cookware items in the accessed image and the at least one subsequent image, having a lowest cost-function cost of all possible matches between cookware items and (2) identifying as a removed cookware item each cookware item that is not present in the particular match having the lowest cost-function cost.
Voigtlaender discloses: associating, in real time, a unique identifier with each identified item in the at least one subsequent image, wherein each unique identifier distinguishes its associated item from every other item in the subsequent image (Voigtlaender: 5. Method: “In order to be able to associate detections over time, we extend Mask R-CNN by an association head which is a fully connected layer that gets region proposals as inputs and predicts an association vector for each proposal…Each association vector represents the identity of a car or a person. They are trained in a way that vectors belonging to the same instance are close to each other and vectors belonging to different instances are far away from each other.”);
identifying each item in the accessed image as one of (1) a removed item or (2) an item in the at least one subsequent image (Voigtlaender: 5. Method: “In order to produce the final result, we still need to decide which detections to report and how to link them into tracks over time. For this, we extend existing tracks with new detections based on their association vector similarity to the most recent detection in that track.”), based on (1) a particular match, between items in the accessed image and the at least one subsequent image, having a lowest cost-function cost of all possible matches between items and (2) identifying as a removed item each item that is not present in the particular match having the lowest cost-function cost (Voigtlaender: 5. Method: “for each class and each frame t, we link together detections at the current frame that have detector confidence larger than a threshold γ with detections selected in the previous frames using the association vector distances from Eq. 7. We only choose the most recent detection for tracks from up to a threshold of β frames in the past. Matching is done with the Hungarian algorithm, while only allowing pairs of detections with a distance smaller than a threshold δ.”; Wherein object track not extended based on object detections in the current track constitute removed objects.);
identifying each item in the at least one subsequent image as one of (1) a newly introduced item or (2) an item in the accessed image (Voigtlaender: 5. Method: “for each class and each frame t, we link together detections at the current frame that have detector confidence larger than a threshold γ with detections selected in the previous frames using the association vector distances from Eq. 7. We only choose the most recent detection for tracks from up to a threshold of β frames in the past. Matching is done with the Hungarian algorithm, while only allowing pairs of detections with a distance smaller than a threshold δ. Finally, all unassigned high confidence detections start new tracks”; Wherein objects detected in the subsequent images are either matched to previously detected objects, or added as new object detections.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the cookware item detection and tracking disclosed by Knighton by implementing the TrackR-CNN for Multi-Object tracking disclosed by Voigtlaender. The suggestion/motivation for doing so would have been “In this paper, we therefore propose to extend the well-known multi-object tracking task to instance segmentation tracking. We call this new task “Multi-Object Tracking and Segmentation (MOTS)”…MOTS requires combining temporal and mask cues for success. We thus propose TrackR-CNN as a baseline method which addresses all aspects of the MOTS task” (Voigtlaender: 1. Introduction). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Knighton with Voigtlaender to obtain the invention as specified in claim 1.
Regarding claim 5, Knighton in view of Voigtlaender discloses: The method of Claim 1, further comprising: creating, based on the identified one or more cookware items from the accessed image, a prior list of previous cookware items present in the field of view; and creating, based on at least one of the one or more subsequent images, a current list of current cookware items present in the field of view (Voigtlaender: 5. Method: “More precisely, for each class and each frame t, we link together detections at the current frame that have detector confidence larger than a threshold γ with detections selected in the previous frames using the association vector distances from Eq. 7. We only choose the most recent detection for tracks from up to a threshold of β frames in the past. Matching is done with the Hungarian algorithm, while only allowing pairs of detections with a distance smaller than a threshold δ. Finally, all unassigned high confidence detections start new tracks.”; Wherein the object tracks and current frame detections constitute the prior list and current list, respectively.).
Regarding claim 6, Knighton in view of Voigtlaender discloses: The method of Claim 5, wherein the cookware items comprise pots (Knighton: 0083: “in embodiments one or more of the cooking assistant components 212 detect individual pots and pans”), the method further comprising: when a number of pots in the prior list is equal to a number of pots in the current list, then matching each pot in the current list to a pot in the prior list (Voigtlaender: 5. Method: “we extend existing tracks with new detections based on their association vector similarity to the most recent detection in that track. More precisely, for each class and each frame t, we link together detections at the current frame that have detector confidence larger than a threshold γ with detections selected in the previous frames using the association vector distances from Eq. 7.”; Wherein all current item detections are matched to previous items detections.).
Regarding claim 7, Knighton in view of Voigtlaender discloses: The method of Claim 6, wherein the pot matching comprises determining a set of matches that has the lowest cost according to a cost function (Voigtlaender: 5. Method: “we link together detections at the current frame that have detector confidence larger than a threshold γ with detections selected in the previous frames using the association vector distances from Eq. 7…Matching is done with the Hungarian algorithm, while only allowing pairs of detections with a distance smaller than a threshold δ.”).
Regarding claim 10, Knighton in view of Voigtlaender discloses: The method of Claim 5, wherein the cookware items comprise pots (Knighton: 0083: “in embodiments one or more of the cooking assistant components 212 detect individual pots and pans”), the method further comprising: when a number of pots in the prior list is less than a number of pots in the current list, then matching each pot in the prior list to a pot in the current list (Voigtlaender: 5. Method: “we extend existing tracks with new detections based on their association vector similarity to the most recent detection in that track. More precisely, for each class and each frame t, we link together detections at the current frame that have detector confidence larger than a threshold γ with detections selected in the previous frames using the association vector distances from Eq. 7.”; Wherein all current item detections are matched to previous items detections.); and for each pot in the current list that is not matched to a pot in the prior list, then adding that unmatched pot to a list of new pots (Voigtlaender: 5. Method: “Finally, all unassigned high confidence detections start new tracks.”; Wherein the creation of new tracks for unmatched current pot detections constitutes the list of new pots.).
Regarding claim 12, Knighton in view of Voigtlaender discloses: The method of Claim 1, further comprising: accessing a temporal sequence of images of the cooking surface (Knighton: 0035: “During a food preparation session, the sensory array 209 collects sensory data for analysis by one or more of the sensory data processing components 206…including, for example one or more thermal sensor(s) 210 (e.g., visible light camera(s) and/or one or more visible light sensor(s) 211 (e.g., visible light camera(s)).”); determining, based on the temporal sequence, whether a cooking event has occurred (Knighton: 0061: “The accessory device 201 can detect (e.g., via sensory array 209) when food, liquid, seasoning, and so forth are added to the cooking environment.”).
Regarding claim 13, Knighton in view of Voigtlaender discloses: The method of Claim 12, further comprising: in response to a determination that a cooking event has occurred, then logging the event in an automatically generated recipe (Knighton: 0066: “recording a recipe creation/generation event (or perhaps an event in which an existing recipe is being followed) utilizes the intelligent cooking assistant architecture 200 to record audio, video, timing, and sensor information in a synchronized data format for an entirety of a food preparation session. In embodiments, the session is initiated and terminated using user input (e.g., such as selection of a UI button or voice command), or using automated event detection (e.g., the system begin recording upon detection of preparations made for cooking). Once a digital recipe is created, it can be stored in recipes 214.”; Wherein the automated event detection is able to determine cooking events and log them).
Regarding claim 14, Knighton in view of Voigtlaender discloses: The method of Claim 13, further comprising: determining, for each event, at least one of a timestamp of the event or a duration of the event; and logging, with the event in the automatically generated recipe, the determined timestamp of the event or the determined duration of the event, or both (Knighton: 0066: “recording a recipe creation/generation event (or perhaps an event in which an existing recipe is being followed) utilizes the intelligent cooking assistant architecture 200 to record audio, video, timing, and sensor information in a synchronized data format for an entirety of a food preparation session. In embodiments, the session is initiated and terminated using user input (e.g., such as selection of a UI button or voice command), or using automated event detection (e.g., the system begin recording upon detection of preparations made for cooking). Once a digital recipe is created, it can be stored in recipes 214.”;
0069: “editing allows for the option to show all timing data to the user, and allow timing to be modified (e.g., when items were added, and how long they cooked).”; Wherein the timing information discloses both timestamps of the cooking event and their duration).
Regarding claim 15, Knighton in view of Voigtlaender discloses: The method of Claim 12, further comprising providing feedback to a user regarding a comparison of a detected cooking event to an expected cooking event, wherein the expected cooking event is an event from the recipe (Knighton: 0112: “User interface 400g also includes the temperature graph 409, now showing two historical and current temperatures—one from the recorded recipe (i.e., using a broken line and italics) and one from the current food preparation session (i.e., using a solid line and non-italics).”).
Regarding claim 16, Knighton in view of Voigtlaender discloses: The method of Claim 12, further comprising providing feedback to a user regarding a detected cooking event (Knighton: 0113: “example user interface 400h shows that recipe step 417a for preheating the pan has completed, and that the user interface 400h has automatically advanced to recipe step 417b”).
Regarding claim 17, Knighton in view of Voigtlaender discloses: The method of Claim 16, wherein the feedback comprises an identification of the detected cooking event in the context of the plurality of steps of the recipe (Knighton: 0113: “example user interface 400h shows that recipe step 417a for preheating the pan has completed, and that the user interface 400h has automatically advanced to recipe step 417b (which is nearing completion) for adding oil to the pan (e.g., when the oil reaches sufficient temperature to proceed to recipe step 417c of adding an egg to the pan).”; Wherein the cooking event is tracked to completion and the user is displayed the next step in the process).
Regarding claim 18, Knighton in view of Voigtlaender discloses: The method of Claim 16, wherein the feedback comprises: an instruction to adjust one or more of the cookware items; or an instruction to adjust a setting associated with the cooking surface (Knighton: 0076: “if the recipe called for cooking a steak for 4 minutes at 450 degrees before flipping, and the current pan is at only 400 degrees, the recipe may be auto adjusted to say that the user should wait 4 minutes and 30 seconds before flipping. In some embodiments, preference is given first to adjust time, and secondly to inform the chef to adjust the cooking surface temperature.”; Wherein the feedback is for the cookware’s temperature to be adjusted and settings of the cooking surface).
As per claim(s) 19 and 20, arguments made in rejecting claim(s) 1 are analogous, respectively. In addition, paragraphs 0119-0120 of Knighton disclose a system comprising a non-volatile computer-readable storage medium storing instructions that are executable by one or more processors.
Claim(s) 2-4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Knighton in view of Voigtlaender, and further in view of Hassani et al. (US 20230169630 A1) hereinafter referenced as Hassani.
Regarding claim 2, Knighton in view of Voigtlaender discloses: The method of Claim 1, further comprising: accessing an optical image comprising an RGB image or grayscale image of the cooking surface (Knighton: 0040: “the visible light sensor(s) 211 include one or more visible light red, green, blue (RGB) cameras, one or more monochromatic cameras, or any other type of visible light camera(s)”);
accessing a thermal image of the cooking surface (Knighton: 0036: “Sensor data from the thermal sensor(s) 210 provides visibility into the surface temperature of a food preparation surface, and/or objects associated therewith, during a food preparation session.”);
and aligning the optical image and the thermal image to produce the accessed image of the cooking surface (Knighton: 0057: “In embodiments one or more of the presentation components 213 implement an “overlay composite” module, which use augmented reality (AR) techniques to overlay a thermal image (e.g., derived from the thermal sensor(s) 210) on top of a visible image (e.g., captured by the visible light sensor(s) 211) to thereby create a composite image having multiple layers”; 0058: “one or more of the cooking assistant components 212 implement object detection (e.g., in connection with one or more of the sensory data analysis components 208 and/or one or more of the ML engines 205),”; Wherein the presentation component is within the Cooking Assistant component).
Knighton does not disclose expressly: removing distortion in each of the optical image and the thermal image.
Hassani discloses: removing distortion in each of the optical image (Hassani: 0023: “At operation 208, the camera compensation service 106 corrects detected fisheye distortion to convert the image into a rectilinear image.”) and the thermal image (Hassani: 0025: “At operation 212, the camera compensation service 106 corrects thermal distortion.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the lens of the visible light camera disclosed by Knighton in view of Voigtlaender with the RGB camera’s fish eye lens disclosed by Hassani, as well as process the optical and thermal images present in Knighton in view of Voigtlaender with the distortion correction algorithm taught by Hassani. The suggestion/motivation for doing so would have been the fisheye lens allows for a wider field of view compared to traditional camera lens and “some images 202 may evidence thermal distortion waves that occurs when rising heat waves cause changes in refractive index or otherwise distort the path of light to the image sensor 102.” (Hassani: 0021; Wherein images may be subjected to thermal distortion due to heat waves). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Knighton in view of Voigtlaender with Hassani to obtain the invention as specified in claim 2.
Regarding claim 3, Knighton in view of Voigtlaender and Hassani discloses: The method of Claim 2, further comprising: detecting one or more features in the accessed image of the cooking surface (Knighton: 0093: “act 301 comprises collecting, using a sensory array, sensor data associated with at least one of, (i) a thermal property of at least one of a food preparation surface or an object on the food preparation surface as observed by at least one thermal sensor, or (ii) a visual property of at least one of the food preparation surface or the object as observed by at least one visible light sensor.”)
(Voigtlaender: 5. Method: “In order to be able to associate detections over time, we extend Mask R-CNN by an association head which is a fully connected layer that gets region proposals as inputs and predicts an association vector for each proposal...Each association vector represents the identity of a car or a person. They are trained in a way that vectors belonging to the same instance are close to each other and vectors belonging to different instances are far away from each other.”; Wherein thermal and visual properties are extracted from detected cookware items.);
and identifying, based on the detected image features, the one or more cookware items (Knighton: 0094: “act 302 comprises determining…or (ii) based at least on the visual property, at least one of an identity or a physical property of the food preparation surface or the object.”)
(Voigtlaender: 5. Method: “for each class and each frame t, we link together detections at the current frame that have detector confidence larger than a threshold γ with detections selected in the previous frames using the association vector distances from Eq. 7.”);
and associating with the unique identifier the location of the cookware item relative to the cooking surface (Knighton: 0058: “one or more of the cooking assistant components 212 implement object detection…which uses both the thermal sensor(s) 210 and the visible light sensor(s) 211 to identify changes that are happening to the food preparation surface. This includes triggering operations in response to certain timing references or timing conditions. In embodiments, object detection also includes the ability to track objects on a per burner basis.”)
(Voigtlaender: Abstract: “This paper extends the popular task of multi-object tracking to multi-object tracking and segmentation (MOTS). Towards this goal, we create dense pixel-level annotations for two existing tracking datasets using a semi-automatic annotation procedure…we propose a new baseline method which jointly addresses detection, tracking, and segmentation with a single convolutional network.”; Wherein each uniquely tracked cookware item is masked and tracked across image frames.).
Regarding claim 4, Knighton in view of Voigtlaender and Hassani discloses: The method of Claim 3, further comprising associating with the unique identifier a size of the cookware item (Knighton: 0044: “the distancing sensor is able to detect one or more of a vertical height of the sensory array 209 with respect to a food preparation surface, a size of the cookware being used (e.g., a 7 inch pan or a 10 inch pan)”;)
(Voigtlaender: Figure 3; 5. Method: “for each class and each frame t, we link together detections at the current frame that have detector confidence larger than a threshold γ with detections selected in the previous frames using the association vector distances from Eq. 7.”; Wherein each cookware item’s size is associated to its classification.).
Claim(s) 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over Knighton in view of Voigtlaender, and further in view of Kato et al. (US 20060208070 A1) hereinafter referenced as Kato.
Regarding claim 8, Knighton in view of Voigtlaender discloses: The method of Claim 5, wherein the cookware items comprise pots (Knighton: 0083: “in embodiments one or more of the cooking assistant components 212 detect individual pots and pans”), the method further comprising: when a number of pots in the current list is less than a number of pots in the prior list, then matching each pot in the current list to a pot in the prior list (Voigtlaender: 5. Method: “we extend existing tracks with new detections based on their association vector similarity to the most recent detection in that track. More precisely, for each class and each frame t, we link together detections at the current frame that have detector confidence larger than a threshold γ with detections selected in the previous frames using the association vector distances from Eq. 7.”; Wherein all current item detections are matched to previous items detections.).
Knighton in view of Voigtlaender does not disclose expressly: for each pot in the prior list that is not matched to a pot in the current list, then adding that unmatched pot to a list of removed pots.
Kato discloses: when a number of items in a current list is less than a number of items in a prior list, then matching each item in the current list to an item in the prior list; and for each item in the prior list that is not matched to an item in the current list, then adding that unmatched item to a list of removed items (Kato: 0030-0031: “The RFID reader 10 periodically scans for nearby RFID tags and produces a list 5 of product identifications (PID) for the filter 30 during each periodic scan. The filter stores the list, and determines if a previously read product ID on a previous list is missing from the current list. If an ID is missing, then the logger 31 is notified, and the PID, RID, and state are recorded. The state of the product is now OUT. A time stamp (TS) is also recorded. Thus, it can be determined how long a product is out of the display area, while the product is examined by the consumer. For example, products X, Y and Z are in the display area. The current product list contains IDs for products X, Y and Z. If a consumer removes product X for examination, the next product list only includes IDs Y and Z. The filtering step 30 detects the removal of the product X and notifies the logging step 31.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the algorithms for logging the removal of previously detected items disclosed by Kato for the tracking of cookware items disclosed by Knighton in view of Voigtlaender. The suggestion/motivation for doing so would have been “the system detects how consumers interact with products” (Kato: 0023; Wherein the system is able to keep track of what items the user has interacted with and currently is interacting with). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Knighton in view of Voigtlaender with Kato to obtain the invention as specified in claim 8.
Regarding claim 9, Knighton in view of Voigtlaender and Kato discloses: The method of Claim 8, wherein matching each pot in the current list to a pot in the prior list comprises determining a set of matches that has the lowest cost according to a cost function (Voigtlaender: 5. Method: “we link together detections at the current frame that have detector confidence larger than a threshold γ with detections selected in the previous frames using the association vector distances from Eq. 7…Matching is done with the Hungarian algorithm, while only allowing pairs of detections with a distance smaller than a threshold δ.”).
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Knighton in view of Voigtlaender, and further in view of Wojke et al. (SIMPLE ONLINE AND REALTIME TRACKING WITH A DEEP ASSOCIATION METRIC) hereinafter referenced as Wojke.
Regarding claim 11, Knighton in view of Voigtlaender discloses: The method of Claim 10, further comprising, for each currently detected pot: comparing that pot to each pot in a prior list of pots to determine whether to match that pot to a pot in the prior list of pots (Voigtlaender: 5. Method: “we extend existing tracks with new detections based on their association vector similarity to the most recent detection in that track. More precisely, for each class and each frame t, we link together detections at the current frame that have detector confidence larger than a threshold γ with detections selected in the previous frames using the association vector distances from Eq. 7.”; Wherein all current item detections are matched to previous items detections.);
and in response to a determination that the pot matches a pot in the prior list of pots, then matching that pot to a list of all pots detected during a cooking episode (Voigtlaender: 5. Method: “we extend existing tracks with new detections based on their association vector similarity to the most recent detection in that track. More precisely, for each class and each frame t, we link together detections at the current frame that have detector confidence larger than a threshold γ with detections selected in the previous frames using the association vector distances from Eq. 7... Matching is done with the Hungarian algorithm, while only allowing pairs of detections with a distance smaller than a threshold δ.”; Wherein matched pots are added to the tracks of previously detected pots.); or in response to a determination that the pot does not match a pot in the prior list of pots, then adding an identification of the pot to the list of all pots detected during the cooking episode (Voigtlaender: 5. Method: “Finally, all unassigned high confidence detections start new tracks.”; Wherein unmatched pots create new tracks, thus adding to the list of previously detected pots.).
Knighton in view of Voigtlaender does not disclose expressly: for each pot in the list of new pots: comparing that pot to each pot in a list of removed pots to determine whether to match that pot to a pot in the list of removed pots; and in response to a determination that the pot matches a pot in the list of removed pots, then matching that pot to a list of all pots detected during a cooking episode; or in response to a determination that the pot does not match a pot in the list of removed pots, then adding an identification of the pot to the list of all pots detected during the cooking episode.
Thus, Knighton in view of Voigtlaender does not disclose expressly, the comparison of unmatched pots to all removed pots, including pots whose previous detection exceeds the frame threshold disclosed by Voigtlaender.
Wojke discloses: an object track matching algorithm which matching current frame object detections with all previously detected objects by introducing a matching cascade. The algorithm iterates through all previously detected objects based on the age of the last frame the object was detected in, prioritizing the detection of more recently detected objects (Wojke: 2.3. Matching Cascade: “Listing 1 outlines our matching algorithm…We then iterate over track age n to solve a linear assignment problem for tracks of increasing age. In line 6 we select the subset of tracks Tn that have not been associated with a detection in the last n frames. In line 7 we solve the linear assignment between tracks in Tn and unmatched detections U. In lines 8 and 9 we update the set of matches and unmatched detections, which we return after completion in line 11. Note that this matching cascade gives priority to tracks of smaller age, i.e., tracks that have been seen more recently.”; Wherein across iterations, the unmatched object tracks constitute the list of removed objects).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the algorithm for matching current object frames to previously detected object tracks disclosed by Wojke for the tracking of cookware pans disclosed by Knighton in view of Voigtlaender. The suggestion/motivation for doing so would have been “Due to this extension, we are able to track through longer periods of occlusion, making SORT a strong competitor to state-of-the-art online tracking algorithms. Yet, the algorithm remains simple to implement and runs in real time” (Wojke: 4. CONCLUSION). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Knighton in view of Voigtlaender with Wojke to obtain the invention as specified in claim 11.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights