Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This communication is in response to the action filed on 06/15/2026.
Claims 1, and 11-12 are currently amended. Claims 13-15 are new. Claims 1-15 are pending.
Response to Arguments
Applicant’s arguments filed on 06/15/2026 on pages 9-14, under REMARKS with respect to 35 U.S.C. 103 have been fully considered but they are not persuasive. Regarding claim 1 applicants on page 11 state that:
PNG
media_image1.png
298
699
media_image1.png
Greyscale
The examiner respectfully disagrees. The examiner would like to point to the primary reference of record US 12,561,840 B1 to GRANSKOG et al, specifically section column 2, line 63-column 3, line 44. The invention of GRANSKOG performs pixel tracing of identified foreground objects and is adapted to as stated at column 3 lines 1-5 “representations of one or more objects in a scene, such as a scene of live gameplay. In at least one embodiment, a rendering engine may output an image of one or more
objects at a first resolution that is to be upscaled to one or more higher output resolutions” and further stating at column 81, lines 18-23 “the computing system utilizes training system 3104 may be used to perform training, deployment, and implementation of machine learning models such as, neural networks, object detection algorithms, computer vision algorithms, etc. for use in deployment system” and finally stating at column 3, lines 25-29 “a super sampling algorithm can utilize a neural network that predicts a blending factor (object color probability vector associated with the area of the scene based on the plurality of foreground objects), for determining an amount by which to weight color values for a current pixel 102 (of one of the plurality of objects) of a current frame 100 and a corresponding historical pixel 112 from a prior warped historical frame”. The examiner would further argue that simply amending language to include multiple tracked objects is an obvious improvement of the embodiment described in GRANSKOG in view of US 11,430,132 B1 to GLASSNER et al., it is also apparent based on the cited sections of GRANSKOG that the prior art does perform multiple foreground object detection and is not limited to single object detection in each scene/frame. GRANSKOG further, since it includes tracking of multiple foreground objects in each subsequent frame/scene would also result in calculating the variation in probabilities for each object resulting in a respective measurement of variability (multiple/a plurality of based on a plurality of objects) among the probabilities indicated by the particular objects color probability vector as recited in the amended limitations.
The applicant further argues regarding that GLASSNER fails to disclose “determining a pixel region in an image frame from the first or the second video sequence that corresponds to an area of the scene using camera parameters, the camera parameters comprising one or more of: pan, tilt, roll or zoom”. The examiner respectfully disagrees and cites GLASSNER figs 5 and 7; column 9, lines 5-54 and specifically states at column 9, lines 20-34 “Between initial camera specifications 510 and final camera specifications 520, the camera moves sideward, rotates around its vertical axis, moves upwards, and rotates around a horizontal axis. It may also change its zoom factor and other relevant configuration parameters. As a result, between perspective 600 and perspective 700, the apparent shapes (as seen in a two-dimensional projection) of the objects and background change, surface colors may shift because the surfaces are viewed from different angles, and small cube 540 becomes concealed behind larger cube 550. While method 100 and 30 method 200 assume fixed camera specifications, they can be extended to support changing camera specifications by transforming frames to correspond to the initial (or any later) camera specifications”, which clearly shows the ability of the camera of the computing system to change field of view/position based on tracking an object in a scene/frame over multiple areas and performs camera operations including zoom, rotation, etc. Please see full rejection to the claims below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 1-15 are rejected under 35 § U.S.C. 103 as being obvious over US 12,561,840 B1 to GRANSKOG et al. (hereinafter “GRANSKOG”) in view of US 11,430,132 B1 to GLASSNER et al. (hereinafter “GLASSNER”).
As per claim 1, GRANSKOG discloses a computer implemented method for determining a color of a tracked object (a computing system and method for tracking objects and determining pixel color values related to said object in the RGB color spectrum; abstract; column 3, line 2 - column 4, line 34), comprising the steps of: providing a first video sequence depicting a scene, the first video sequence comprising a plurality of image frames (the system is adapted to receive image frames from a video sequence recorder which is a camera device, and captures vise of a scene comprising an environment; figs 3-4B and fig 9; column 3, line 2 - column 4, line 65; column 5, line 27-column 6, line 38; column 12, lines 1-56; column 21, lines 1-11); for each image frame of the plurality of image frames, detecting foreground objects (the system is adapted to perform object tracking by tracking object feature points, and detects objects that are in the foreground of the captured images; abstract; column 3, line 2 - column 4, line 34); for each area of a plurality of areas in the scene, analyzing the first video sequence by detecting a plurality of foreground objects in the area of the scene and calculating an object color probability vector associated with the area of the scene based on the plurality of foreground objects (a computing system performs an upscaling process (analyzing the first video sequence) on an input video comprising representations of one or more objects in a scene (the scene including multiple regions of interest/areas), such as a scene of live gameplay a rendering engine outputs an image of one or more objects at a first resolution that is to be upscaled to one or more higher output resolutions between a current frame 100 and at least one prior or historical frame 110 in a sequence (each frame depicting an area of the scene), in order to ensure that this blending occurs for corresponding points on objects in these frames, this prior historical color data can be warped based upon motion detected between this historical frame 110 and this current frame, such as may be indicated by a set of motion vectors output from this rendering engine, or otherwise determined, such warping can ensure that points, such as feature points of the plurality of foreground objects, for various images are tracked over time and corresponding color values (color probability vector) used for blending, which can help reduce a presence of artifacts such as noise or flickering during playback; column 2, line 63-column 3, line 44), wherein each value in the object color probability vector relates to a color from a set of predefined colors and indicates a probability that a foreground object located in the area of the scene has the color (the computing system is adapted to for each region depicted in the scene comprising tracked objects determine a motion vector and probability value for a target pixel of a tracked object and further can include generating adjusted weightings between current and previous color values for pixel locations that are determined to correspond to motion vector difference regions that have a probability of image artifacts/objects; abstract; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 12, lines 1-56); for each area, calculating a measurement of variability among the probabilities indicated by the object color probability vector associated with the area of the scene, and associating the measurement of variability with the area of the scene (the computing system is adapted to find a value relating to the variation in probabilities and can be shown as a Gaussian filtering ratio for each region and the variable is expressed as a ratio for example a large ratio e.g., 9x up sampling process can result in a relatively jagged or low resolution appearing image, as there will in general be three pixel by three pixel blocks that share a same color value, further a down sampling filter can be applied, such as may involve 2x2 down sampling with a regular box filter, although other filters and ratios can be utilized as well; abstract; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 12, lines 1-56); providing a second video sequence depicting the scene (the camera is adapted to capture video and image frame sequences and would be adapted to capture a second video sequence subsequently following the first; column 3, lines 2-64; column 4, lines 8-65; column 48, lines 52-67; claim 1); tracking a foreground object in the second video sequence, the tracked foreground object being located in a first area of the plurality of areas in the scene in a first image frame of the second video sequence, and in a second different area of the plurality of areas in the scene in a second image frame of the second video sequence (the system is adapted to perform object tracking by tracking object feature points, and detects objects that are in the foreground of the captured images in the first and second video sequence depicting the regions of interest; abstract; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 12, lines 1-56; column 34, lines 1-59); determining a first set of colors of the tracked foreground object in the first image frame, and determining a second different set of colors of the tracked foreground object in the second image frame (the computing system is adapted to track an object and determine its colors in the RGB spectrum and is adapted to track a first color of the objects pixel and a second set of colors of an objects pixel wherein the colors differ; figs 13-14B; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 11, lines 3-19; column 12, lines 1-56; column 34, lines 1-59; column 35, lines 59-column 36 line 64); and upon determining that the measurement of variability associated with the first area of the scene is lower than the measurement of variability associated with the second area of the scene, determining that the color(s) of the tracked foreground object is the first set of colors, and otherwise determining that the color(s) of the tracked foreground object is the second set of colors (a process 460 for generating an image can be performed as illustrated in FIG. 4C. In at least one embodiment, one or more temporal variations of a first color of one or more pixels can be determined at step 462 of the tracked object in the first and second video sequence, and can also further determine using temporal variations can be provided 464 as input to one or more neural networks, along with color data and other feature inputs and provides one or more second colors of these one or more pixels can be determined 466 using these one or more neural networks based, adapted for color recognition; figs 3-4C, 9, and 13-14B; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 11, lines 3-19; column 12, lines 1-56; column 34, lines 1-59), wherein the first and second video sequence is captured by a camera with a same field of view in the scene, wherein an area of the scene corresponds to a same pixel region in the image frames of the first and second video sequence (the computing system includes a fixed position video camera which has the same field of view when capturing all video frames and video sequences; fig 9; column 9, line 15-column 10, line 57; column 21, lines 1-60; column 23, lines 1-43; column 39, lines 8-53). GRANSKOG fails to disclose or wherein the first and second video sequence is captured by a camera with a changing field of view in the scene, wherein the method further comprises the step of: determining a pixel region in an image frame from the first or the second video sequence that corresponds to an area of the scene using camera parameters, the camera parameters comprising one or more of: pan, tilt, roll or zoom.
GLASSNER discloses or wherein the first and second video sequence is captured by a camera with a changing field of view in the scene, wherein the method further comprises the step of: determining a pixel region in an image frame from the first or the second video sequence that corresponds to an area of the scene using camera parameters, the camera parameters comprising one or more of: pan, tilt, roll or zoom (the computing system adapted for pixel color determination is adapted to capture multiple frames of video sequences in order to track an object and is adapted to perform camera parameter/feature adjustments which means zooming and rotating of camera view to better see pixels of the objects of interest; figs 5-6, 8, 9; column 5, lines 1-2; column 9, lines 5-64; column 10, lines 63-67).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify GRANSKOG to have camera parameters comprising one or more of: pan, tilt, roll or zoom of GLASSNER reference. The Suggestion/motivation for doing so would have been to provide camera parameters so that the camera may move and or zoom in order to more effectively track objects frame to frame by providing adjustable field of vision as suggested by column 9, lines 5-54 of GLASSNER. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine GLASSNER with GRANSKOG to obtain the invention as specified in claim 1.
As per claim 2, GRANSKOG in view of GLASSNER discloses the method of claim 1. Modified GRANSKOG further discloses further comprising the steps of: labelling all occurrences of the tracked foreground object in the second video sequence with the determined color(s) (the object that is being tracked has its pixels assigned a pixel value in the RGB spectrum for color value which acts as a label in order to tell the user what color within RGB the pixel represents; column 2, line 63-column 3, line 66; column 4, lines 4-22; column 10, lines 13-57).
As per claim 3, GRANSKOG in view of GLASSNER discloses the method of claim 1. Modified GRANSKOG further discloses wherein each of the first and second set of colors comprises one of: a single color value (the computing system is adapted to compute and assign a color value of a single pixel/point can be determined and is assigned as a first color value; abstract; fig 4b; column 3, lines 2-29; column 11, lines 1-53); a plurality of color values (the computing system is further adapted to compute and assign a color value for a plurality of pixels/points is determined in relation to the tracked object region and can apply a second color to said pixel if determined; abstract; abstract; fig 4b; column 3, lines 2-29; column 4, lines 4-22; column 11, lines 1-53); or a plurality of color values, each color value associated with a probability that the tracked object has the color ( and is further adapted to in some circumstances after applying color values to a plurality of pixels generate/adjust weightings between current and previous color values for the plurality of pixel locations that are determined to correspond to motion vector difference regions that have a probability of image artifacts/objects; abstract; figs 3-4B; column 3, lines 2-29; column 4, lines 4-22; column 10, lines 13-57).
As per claim 4, GRANSKOG in view of GLASSNER discloses the method of claim 1. Modified GRANSKOG further discloses wherein each area of the scene corresponds to a single pixel coordinate in the image frames of the first and second video sequence (images of a specific resolution will have pixels grouped based on the resolution and color value will be determined and assigned based on the center pixel of the pixels encompassed by the area determined by the user in the example provided the resolution was up sampled 4X of the captured frames of the first or second video sequence captured via the camera; column 13, lines 3-35).
As per claim 5, GRANSKOG in view of GLASSNER discloses the method of claim 1. Modified GRANSKOG further discloses wherein the step of detecting foreground objects in the image frame comprises: determining a location and an extent of each detected foreground object in the image frame, wherein the location and extent comprises one of: a pixel mask, or a bounding box (in order to identify objects which, comprise pixels of interest the user finds the X,Y position of each point/pixel and finds these using a color variance masking techniques; column 10, lines 13-58; column 11, lines 20-54).
As per claim 6, GRANSKOG in view of GLASSNER discloses the method of claim 1. Modified GRANSKOG further discloses wherein the step of analyzing the first video sequence comprises, for each foreground object located in the area of the scene in an image frame of the first video sequence: determining one or more colors of the foreground object from pixel data depicting the foreground object in the image frame (the tracked object which would be in the foreground of the captured images includes a first color of a set of one or more pixels is found at step 462, and second color of those pixels may be determined at step 466; abstract; figs 4A-C; column 11, lines 3-19); and using the determined one or more colors when calculating the object color probability vector associated with the area of the scene (a neural network can accept both temporal and spatial variations as inputs for determining pixel color values, such as may be used in determining how to weight two or more color values to be blended for the particular set of pixels and includes a probability of image artifacts; abstract; figs 3-4B; column 3, lines 2-29; column 4, lines 4-22; column 10, lines 13-57; column 11, lines 3-19).
As per claim 7, GRANSKOG in view of GLASSNER discloses the method of claim 1. Modified GRANSKOG further discloses wherein the step of analyzing the first video sequence comprises, for each foreground object located in the area of the scene in an image frame of the first video sequence: receiving a plurality of color values, each color value associated with a probability that the foreground object has the color in the image frame (a neural network can accept both temporal and spatial variations as inputs for determining pixel color values, such as may be used in determining how to weight two or more color values to be blended for the particular set of pixels and includes a probability of image artifacts; abstract; figs 3-4B; column 3, lines 2-29; column 4, lines 4-22; column 10, lines 13-57; column 11, lines 3-19); and using the plurality of color values and their associated probabilities when calculating the object color probability vector associated with the area of the scene (the first and second color values of the pixels are considered when determining probability of artifacts in the captured vides image frames; abstract; figs 3-4B; column 3, lines 2-29; column 4, lines 4-22; column 10, lines 13-57; column 11, lines 3-19).
As per claim 8, GRANSKOG in view of GLASSNER discloses the method of claim 1. Modified GRANSKOG further discloses wherein the step of calculating an object color probability vector for an area of the plurality of areas in the scene comprises detecting at least a threshold number of foreground objects in the area of the scene (the computing system has a threshold minimum of one object in the frames to be tracked or the method for tracking object(s) would not be performed if there was not at least one obj3ect and therefore the threshold is one object; column 2, line 63-column 3, line 66).
As per claim 9, GRANSKOG in view of GLASSNER discloses the method of claim 1. Modified GRANSKOG further discloses wherein the first video sequence is captured during a first time period of a day (the camera is adapted to capture video and image frame sequences and would be adapted to capture a first video during a desired first time period determined to be suitable by the user and applied as desired; column 3, lines 2-64; column 4, lines 8-65; column 48, lines 52-67; column 55, lines 12-14; claim 1), wherein the second video sequence is captured during a second time period of a subsequent day, wherein the second time period is entirely encompassed within the first time period (the computing system is further adapted to capture videos and subsequent images or video frames in a sequence and would be programed by the user via camera parameters to capture the second video subsequently a day after the first and may choose to include the second video time period within the first; column 3, lines 2-64; column 4, lines 8-65; column 48, lines 52-67; column 55, lines 12-14; claim 1).
As per claim 10, GRANSKOG in view of GLASSNER discloses the method of claim 9. Modified GRANSKOG further discloses wherein the measurement of variability is at least one of: variance, standard deviation, mean absolute deviation, median absolute deviation or coefficient of variations (the computing system measures variability of the performed method using a plurality of methods including a variance mask, standard deviations, and mean value averaging; column 3, lines 2-44; column 5, lines 26-47).
As per claim 11, GRANSKOG discloses a system comprising: one or more processors (a computing system and method for tracking objects and determining pixel color values related to said object in the RGB color spectrum the computing system comprising a memory and processor component; abstract; column 3, line 2 - column 4, line 34; column 16, lines 3-35); and one or more non-transitory computer-readable media storing first computer executable instructions that, when executed by the one or more processors (the computing system comprising a memory and processor component which store and execute data, programs, and instructions; abstract; column 3, line 2 - column 4, line 34; column 16, lines 3-35), cause the system to perform actions comprising: providing a first video sequence depicting a scene, the first video sequence comprising a plurality of image frames (the system is adapted to receive image frames from a video sequence recorder which is a camera device, and captures vise of a scene comprising an environment; figs 3-4B and fig 9; column 3, line 2 - column 4, line 65; column 5, line 27-column 6, line 38; column 12, lines 1-56; column 21, lines 1-11); for each image frame of the plurality of image frames, detecting foreground objects in the image frame (the system is adapted to perform object tracking by tracking object feature points, and detects objects that are in the foreground of the captured images; abstract; column 3, line 2 - column 4, line 34); for each area of a plurality of areas in the scene, analyzing the first video sequence and calculating an object color probability vector associated with the area of the scene based on the plurality of foreground objects (a computing system performs an upscaling process (analyzing the first video sequence) on an input video comprising representations of one or more objects in a scene (the scene including multiple regions of interest/areas), such as a scene of live gameplay a rendering engine outputs an image of one or more objects at a first resolution that is to be upscaled to one or more higher output resolutions between a current frame 100 and at least one prior or historical frame 110 in a sequence (each frame depicting an area of the scene), in order to ensure that this blending occurs for corresponding points on objects in these frames, this prior historical color data can be warped based upon motion detected between this historical frame 110 and this current frame, such as may be indicated by a set of motion vectors output from this rendering engine, or otherwise determined, such warping can ensure that points, such as feature points of the plurality of foreground objects, for various images are tracked over time and corresponding color values (color probability vector) used for blending, which can help reduce a presence of artifacts such as noise or flickering during playback; column 2, line 63-column 3, line 44), wherein each value in the object color probability vector relates to a color from a set of predefined colors and indicates a probability that a foreground object located in the area of the scene has the color (the computing system is adapted to for each region depicted in the scene comprising tracked objects determine a motion vector and probability value for a target pixel of a tracked object and further can include generating adjusted weightings between current and previous color values for pixel locations that are determined to correspond to motion vector difference regions that have a probability of image artifacts/objects; abstract; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 12, lines 1-56); for each area of the plurality of areas of the scene, calculating a measurement of variability among the probabilities indicated by the object color probability vector associated with the area of the scene, and associating the measurement of variability with the area of the scene (the computing system is adapted to find a value relating to the variation in probabilities and can be shown as a Gaussian filtering ratio for each region and the variable is expressed as a ratio for example a large ratio e.g., 9x up sampling process can result in a relatively jagged or low resolution appearing image, as there will in general be three pixel by three pixel blocks that share a same color value, further a down sampling filter can be applied, such as may involve 2x2 down sampling with a regular box filter, although other filters and ratios can be utilized as well; abstract; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 12, lines 1-56); providing a second video sequence depicting the scene (the camera is adapted to capture video and image frame sequences and would be adapted to capture a second video sequence subsequently following the first; column 3, lines 2-64; column 4, lines 8-65; column 48, lines 52-67; claim 1); tracking a foreground object in the second video sequence, the tracked foreground object being located in a first area of the plurality of areas in the scene in a first image frame of the second video sequence, and in a second different area of the plurality of areas in the scene in second image frame of the second video sequence (the system is adapted to perform object tracking by tracking object feature points, and detects objects that are in the foreground of the captured images in the first and second video sequence depicting the regions of interest; abstract; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 12, lines 1-56; column 34, lines 1-59); determining a first set of colors of the tracked foreground object in the first image frame, and determining a second different set of colors of the tracked foreground object in the second image frame (the computing system is adapted to track an object and determine its colors in the RGB spectrum and is adapted to track a first color of the objects pixel and a second set of colors of an objects pixel wherein the colors differ; figs 13-14B; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 11, lines 3-19; column 12, lines 1-56; column 34, lines 1-59; column 35, lines 59-column 36 line 64); upon determining that the measurement of variability associated with the first area of the scene is lower than the measurement of variability associated with the second area of the scene, determining that the color(s) of the tracked foreground object is the first set of colors, and otherwise determining that the color(s) of the tracked foreground object is the second set of colors (a process 460 for generating an image can be performed as illustrated in FIG. 4C. In at least one embodiment, one or more temporal variations of a first color of one or more pixels can be determined at step 462 of the tracked object in the first and second video sequence, and can also further determine using temporal variations can be provided 464 as input to one or more neural networks, along with color data and other feature inputs and provides one or more second colors of these one or more pixels can be determined 466 using these one or more neural networks based, adapted for color recognition; figs 3-4C, 9, and 13-14B; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 11, lines 3-19; column 12, lines 1-56; column 34, lines 1-59), wherein the first and second video sequence is captured by a camera with a same field of view in the scene, wherein an area of the scene corresponds to a same pixel region in the image frames of the first and second video sequence (the computing system includes a fixed position video camera which has the same field of view when capturing all video frames and video sequences; fig 9; column 9, line 15-column 10, line 57; column 21, lines 1-60; column 23, lines 1-43; column 39, lines 8-53). GRANSKOG fails to disclose or wherein the first and second video sequence is captured by a camera with a changing field of view in the scene, and wherein a pixel region in an image frame from the first or the second video sequence that corresponds to an area of the scene is determined using camera parameters, the camera parameters comprising one or more of: pan, tilt, roll or zoom.
GLASSNER discloses or wherein the first and second video sequence is captured by a camera with a changing field of view in the scene, and wherein a pixel region in an image frame from the first or the second video sequence that corresponds to an area of the scene is determined using camera parameters, the camera parameters comprising one or more of: pan, tilt, roll or zoom (the computing system adapted for pixel color determination is adapted to capture multiple frames of video sequences in order to track an object and is adapted to perform camera parameter/feature adjustments which means zooming and rotating of camera view to better see pixels of the objects of interest; figs 5-6, 8, 9; column 5, lines 1-2; column 9, lines 5-64; column 10, lines 63-67).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify GRANSKOG to have camera parameters comprising one or more of: pan, tilt, roll or zoom of GLASSNER reference. The Suggestion/motivation for doing so would have been to provide camera parameters so that the camera may move and or zoom in order to more effectively track objects frame to frame by providing adjustable field of vision as suggested by column 9, lines 5-54 of GLASSNER. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine GLASSNER with GRANSKOG to obtain the invention as specified in claim 11.
As per claim 12, GRANSKOG discloses a system comprising a color matching system (a computing system and method for tracking objects and determining pixel color values related to said object in the RGB color spectrum the computing system comprising a memory and processor component; abstract; column 3, line 2 - column 4, line 34; column 16, lines 3-35) and a forensic search application (the forensic search application would be integrated using neural network models with adjustable weighted parameters which would be trained to accomplish substantially the same tasks as the forensic search model/system claimed and would be stored in the computing systems memory and executed by its processors; figs 9 and 32; column 12, lines 2-56), wherein the color matching system comprises: one or more processors (the computing system comprising a memory and processor component which store and execute data, programs, and instructions; abstract; column 3, line 2 - column 4, line 34; column 16, lines 3-35); and one or more non-transitory computer-readable media storing first computer executable instructions that, when executed by the one or more processors (the computing system comprising a memory and processor component which store and execute data, programs, and instructions; abstract; column 3, line 2 - column 4, line 34; column 16, lines 3-35), cause the system to perform actions comprising: providing a first video sequence depicting a scene, the first video sequence comprising a plurality of image frames (the system is adapted to receive image frames from a video sequence recorder which is a camera device, and captures vise of a scene comprising an environment; figs 3-4B and fig 9; column 3, line 2 - column 4, line 65; column 5, line 27-column 6, line 38; column 12, lines 1-56; column 21, lines 1-11); for each image frame of the plurality of image frames, detecting foreground objects in the image frame (the system is adapted to perform object tracking by tracking object feature points, and detects objects that are in the foreground of the captured images; abstract; column 3, line 2 - column 4, line 34); for each area of a plurality of areas in the scene, analyzing the first video sequence by detecting a plurality of foreground objects in the area of the scene and calculating an object color probability vector associated with the area of the scene based on the plurality of foreground objects (a computing system performs an upscaling process (analyzing the first video sequence) on an input video comprising representations of one or more objects in a scene (the scene including multiple regions of interest/areas), such as a scene of live gameplay a rendering engine outputs an image of one or more objects at a first resolution that is to be upscaled to one or more higher output resolutions between a current frame 100 and at least one prior or historical frame 110 in a sequence (each frame depicting an area of the scene), in order to ensure that this blending occurs for corresponding points on objects in these frames, this prior historical color data can be warped based upon motion detected between this historical frame 110 and this current frame, such as may be indicated by a set of motion vectors output from this rendering engine, or otherwise determined, such warping can ensure that points, such as feature points of the plurality of foreground objects, for various images are tracked over time and corresponding color values (color probability vector) used for blending, which can help reduce a presence of artifacts such as noise or flickering during playback; column 2, line 63-column 3, line 44), wherein each value in the object color probability vector relates to a color from a set of predefined colors and indicates a probability that a foreground object located in the area of the scene has the color (the computing system is adapted to for each region depicted in the scene comprising tracked objects determine a motion vector and probability value for a target pixel of a tracked object and further can include generating adjusted weightings between current and previous color values for pixel locations that are determined to correspond to motion vector difference regions that have a probability of image artifacts/objects; abstract; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 12, lines 1-56); for each area of the plurality of areas of the scene, calculating a measurement of variability among the probabilities indicated by the object color probability vector associated with the area of the scene, an associating the measurement of variability with the area of the scene (the computing system is adapted to find a value relating to the variation in probabilities and can be shown as a Gaussian filtering ratio for each region and the variable is expressed as a ratio for example a large ratio e.g., 9x up sampling process can result in a relatively jagged or low resolution appearing image, as there will in general be three pixel by three pixel blocks that share a same color value, further a down sampling filter can be applied, such as may involve 2x2 down sampling with a regular box filter, although other filters and ratios can be utilized as well; abstract; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 12, lines 1-56); providing a second video sequence depicting the scene (the camera is adapted to capture video and image frame sequences and would be adapted to capture a second video sequence subsequently following the first; column 3, lines 2-64; column 4, lines 8-65; column 48, lines 52-67; claim 1); providing a second video sequence depicting the scene (the camera is adapted to capture video and image frame sequences and would be adapted to capture a second video sequence subsequently following the first; column 3, lines 2-64; column 4, lines 8-65; column 48, lines 52-67; claim 1); tracking a foreground object in the second video sequence, the tracked foreground object being located in a first area of the plurality of areas in the scene in a first image frame of the second video sequence, and in a second different area of the plurality of areas in the scene in second image frame of the second video sequence (the system is adapted to perform object tracking by tracking object feature points, and detects objects that are in the foreground of the captured images in the first and second video sequence depicting the regions of interest; abstract; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 12, lines 1-56; column 34, lines 1-59); determining a first set of colors of the tracked foreground object in the first image frame, and determining a second different set of colors of the tracked foreground object in the second image frame (the computing system is adapted to track an object and determine its colors in the RGB spectrum and is adapted to track a first color of the objects pixel and a second set of colors of an objects pixel wherein the colors differ; figs 13-14B; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 11, lines 3-19; column 12, lines 1-56; column 34, lines 1-59; column 35, lines 59-column 36 line 64); upon determining that the measurement of variability associated with the first area of the scene is lower than the measurement of variability associated with the second area of the scene, determining that the color(s) of the tracked foreground object is the first set of colors, and otherwise determining that the color(s) of the tracked foreground object is the second set of colors (a process 460 for generating an image can be performed as illustrated in FIG. 4C. In at least one embodiment, one or more temporal variations of a first color of one or more pixels can be determined at step 462 of the tracked object in the first and second video sequence, and can also further determine using temporal variations can be provided 464 as input to one or more neural networks, along with color data and other feature inputs and provides one or more second colors of these one or more pixels can be determined 466 using these one or more neural networks based, adapted for color recognition; figs 3-4C, 9, and 13-14B; column 3, line 2 - column 4, line 34; column 5, line 27-column 6, line 38; column 11, lines 3-19; column 12, lines 1-56; column 34, lines 1-59); receiving a search request comprising a first color value (assign a color value of a single pixel/point can be determined and is assigned as a first color value; abstract; fig 4b; column 3, lines 2-29; column 10, lines 13-57; column 11, lines 1-53); determining that the first color value matches a color determined for the foreground object (the color value is compared to a second color feature detected of the pixels and neural network models are used to “blend” the color to arrive at the true color determination for the tracked object and may have color filter applied to further alter color; abstract; fig 4b; column 3, lines 2-29; column 10, lines 13-57; column 11, lines 1-53); and returning a search response based at least in part on the foreground object (abstract; fig 4b; column 3, lines 2-29; column 10, lines 13-57; column 11, lines 1-53); wherein the forensic search application comprises: one or more processors (the forensic search application would be integrated using neural network models with adjustable weighted parameters which would be trained to accomplish substantially the same tasks as the forensic search model/system claimed and would be stored in the computing systems memory and executed by its processors; figs 9 and 32; column 12, lines 2-56); and one or more non-transitory computer-readable media storing second computer executable instructions that, when executed by the one or more processors (the computing system comprising a memory and processor component which store and execute data, programs, and instructions; abstract; column 3, line 2 - column 4, line 34; column 16, lines 3-35), cause the forensic search application to perform actions comprising: providing the search request comprising the first color value to the color matching system (to extract a first color value from the first tracked object of the first image frames of the captured video sequence based on the determination and input of a frame comprising an object to be tracked; abstract; fig 4b; column 3, lines 2-29; column 10, lines 13-57; column 11, lines 1-53); receiving a search response from the color matching system (finding a color match based on analyzing the pixels and pulling two color features and performing a weighted blending operation performed via neural network models which substantially act like a forensics model and are ran operated and stored on the computing system; abstract; fig 4b; column 3, lines 2-29; column 10, lines 13-57; column 11, lines 1-53); and displaying data from the search response to a user (the data is output to a display and user interface to be viewed by the user; column 38, lines 35-54), wherein the first and second video sequence is captured by a camera with a same field of view in the scene, wherein an area of the scene corresponds to a same pixel region in the image frames of the first and second video sequence (the computing system includes a fixed position video camera which has the same field of view when capturing all video frames and video sequences; fig 9; column 9, line 15-column 10, line 57; column 21, lines 1-60; column 23, lines 1-43; column 39, lines 8-53). GRANSKOG fails to disclose or wherein the first and second video sequence is captured by a camera with a changing field of view in the scene, and wherein a pixel region in an image frame from the first or the second video sequence that corresponds to an area of the scene is determined using camera parameters, the camera parameters comprising one or more of: pan, tilt, roll or zoom.
GLASSNER discloses or wherein the first and second video sequence is captured by a camera with a changing field of view in the scene, and wherein a pixel region in an image frame from the first or the second video sequence that corresponds to an area of the scene is determined using camera parameters, the camera parameters comprising one or more of: pan, tilt, roll or zoom (the computing system adapted for pixel color determination is adapted to capture multiple frames of video sequences in order to track an object and is adapted to perform camera parameter/feature adjustments which means zooming and rotating of camera view to better see pixels of the objects of interest; figs 5-6, 8, 9; column 5, lines 1-2; column 9, lines 5-64; column 10, lines 63-67).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify GRANSKOG to have camera parameters comprising one or more of: pan, tilt, roll or zoom of GLASSNER reference. The Suggestion/motivation for doing so would have been to provide camera parameters so that the camera may move and or zoom in order to more effectively track objects frame to frame by providing adjustable field of vision as suggested by column 9, lines 5-54 of GLASSNER. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine GLASSNER with GRANSKOG to obtain the invention as specified in claim 12.
As per claim 13, GRANSKOG in view of GLASSNER discloses the method of claim 1. Modified GRANSKOG further discloses wherein detecting foreground objects comprises using a neural network-based object detector (the computing system utilizes training system 3104 may be used to perform training, deployment, and implementation of machine learning models such as, neural networks, object detection algorithms, computer vision algorithms, etc. for use in deployment system; column 3, lines 1-44; column 81, lines 18-23).
As per claim 14, GRANSKOG in view of GLASSNER discloses the method of claim 1. Modified GRANSKOG further discloses wherein determining colors of the foreground objects comprises using a neural network (the computing system comprises a super sampling algorithm can utilize a neural network that predicts a blending factor (determining colors), for determining an amount by which to weight color values for a current pixel 102 of a current frame 100 and a corresponding historical pixel 112 from a prior warped historical frame; column 3, lines 1-44).
As per claim 15, GRANSKOG in view of GLASSNER discloses the method of claim 1. Modified GRANSKOG further discloses wherein the foreground objects comprise a plurality of foreground object classes (the computing system is adapted to use a CNN to sort detected objects into different classes wherein the detected objects are fore ground objects of the images/scenes being analyzed; column 81, line 40-column 82, line 67; column 92, lines 8-35).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Examiner's Note: Examiner has cited figures, and paragraphs in the references as applied to the claims above for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested for the applicant, in preparing the responses, to fully consider the references in entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. Examiner has also cited references in PTO892 but not relied on, which are relevant and pertinent to the applicant’s disclosure, and may also be reading (anticipatory/obvious) on the claims and claimed limitations. Applicant is advised to consider the references in preparing the response/amendments in-order to expedite the prosecution.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEVIN JACOB DHOOGE whose telephone number is (571) 270-0999. The examiner can normally be reached 7:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800- 786-9199 (IN USA OR CANADA) or 571-272-1000.
/D J DHOOGE/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights