DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
2. Receipt is acknowledged of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Information Disclosure Statement
3. The information disclosure statement (IDS) submitted on 05/14/2025. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
4. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
5. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
6. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
7. Claim(s) 1, 2 and 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Messer et al. (US 2022/0020255 A1) in view of Herdade et al. (US 2021/0201044 A1) and Bolle et al. (US 2007/0201694 A1).
8. With reference to claim 1, Messer teaches A management system for camera footage, comprising: a memory device configured to store the camera footage; processing circuitry configured to perform various processing; and a display configured to output an image, image to be output from the display device (“In a particular example the surveillance system may comprise at least one first, say fixed or vehicle mounted, camera apparatus and at least one mobile, say body worn, camera apparatus and the system may be arranged to issue a prompt to the operator of the mobile camera apparatus to seek to acquire a second image/video clip of an object in respect of which an image/video clip has already been captured by the first camera which resulted in a match against a watch list.” [0063] “The camera apparatus may be arranged to record video and store the recorded video on storage means. The camera apparatus may be arranged to allow later play back of recorded video and may be arranged to allow sending of recorded video via the video encoder either as it is played back or otherwise.” [0083] “The complete result combining the result of the analysis, detailed analysis and live video will be presented to the operator as a single output on their display (or separately if desired).” [0231]) Messer also teaches the processing circuitry is configured to: generate recognition information on an object shown in the camera footage by object recognition processing on the camera footage stored in the memory device; (“The camera apparatus may comprise an object recognition module for classifying objects in captured images and/or video clips. Preferably the object recognition module is arranged to process captured image data/video clip data which corresponds to a captured image/captured video clip in which an object of interest has been first detected by the object detection module, to thereby classify objects of interest.” [0026] “The camera apparatus may be arranged to record video and store the recorded video on storage means. … The camera apparatus may be arranged to allow the capture of images and/or video clips from recorded video, say as this is playback at the camera apparatus. Such captured images and/or video clips may then be treated/processed as described above in the same way as images and/or video clips captured when live video is being captured and corresponding object recognition data relating to such captured images and/or video clips may be sent along with the recorded video where this is sent via the video encoder.” [0083-0084]) Messer further teaches when the recognition information on the object include recognition information on a human, generate scene information in which the recognition information on the human in which the human is recognized and store the scene information in the memory device; and perform processing based on the scene information stored in the memory device, (“The camera apparatus may be arranged to record video and store the recorded video on storage means.” [0083] “The server may be arranged to send packages of data to end user devices for allowing end users to view video received from camera apparatus and/or for allowing end users to be informed of object recognition events. The server may be arranged to add meta data which relates to an object that has been recognised to at least some such packages of data. The meta data may include GPS data, image/video analytics data, unique ID data, location data indicating a region of interest in an image—eg of a detected face, time-stamp data, device data, etc.” [0085-0086] “Further the camera apparatus may comprise a face detection module for detecting the presence of faces in said captured images and/or video clips. The object recognition data may comprise captured image and/or video clip data for images and/or video clips in which faces have been detected. The camera apparatus may be arranged to send captured image and/or video clip data as object recognition data where the captured image and/or video clip data corresponds to a captured image and/or video clip in which a face has been detected by the face detection module.” [0097-0099])
PNG
media_image1.png
470
415
media_image1.png
Greyscale
Messer does not explicitly teach generate linguistic information on a scene shown in the camera footage by linguistic processing on the camera footage stored in the memory device; associated with the linguistic information on the scene generated by the linguistic processing on the camera footage; and perform reproduction processing on a scene shown in the camera footage, wherein the reproduction processing comprises; rendering an abstracted image of a space included in the camera footage based on the recognition information on a static object included in the scene information; rendering an abstracted image of the human included in the camera footage on the abstracted image of the space based on the recognition information on the human included in the scene information; generating caption information on the scene based on the linguistic information on the scene included in the scene information; and generating a reproduced image by adding the caption information to the abstracted image of the space in which the abstracted image of the human is rendered. This is what Herdade teaches. Herdade teaches generate linguistic information on a scene shown in the camera footage by linguistic processing on the camera footage stored in the memory device; associated with the linguistic information on the scene generated by the linguistic processing on the camera footage; generating caption information on the scene based on the linguistic information on the scene included in the scene information; and adding the caption information to the image (“The present disclosure provides novel systems and methods for automatic generation of a caption representing content of a digital content item (e.g., a digital image) using deep learning and spatial relationships between detected objects depicted in the digital content item. Examples of object spatial relationship information include relative position and relative size of objects depicted in digital content item.” [0050] “client device 200 includes a processing unit (CPU) 222 in communication with a mass memory 230 via a bus 224. Client device 200 also includes a power supply 226, one or more network interfaces 250, an audio interface 252, a display 254, a keypad 256, an illuminator 258, an input/output interface 260, a haptic interface 262, an optional global positioning systems (GPS) receiver 264 and a camera(s) or other optical, thermal or electromagnetic sensors 266. Device 200 can include one camera/sensor 266, or a plurality of cameras/sensors 266,” [0078] “Audio interface 252 is arranged to produce and receive audio signals such as the sound of a human voice. For example, audio interface 252 may be coupled to a speaker and microphone (not shown) to enable telecommunication with others and/or generate an audio acknowledgement for some action. Display 254 may be a liquid crystal display (LCD), gas plasma, light emitting diode (LED), or any other type of display used with a computing device.” [0081] “Process 400 of FIG. 4 details steps performed in accordance with exemplary embodiments of the present disclosure for automatically generating an image caption reflecting content of a digital content item (e.g., an image) using spatial relationships between detected objects depicted in the digital content item. According to some embodiments, as discussed herein with relation to FIG. 4, the process involves automatically detecting a number of objects in a digital content item and extracting appearance and geometry features corresponding to each detected object. In addition, the process involves using the appearance and geometry features to generate encoded output which is then decoded in order to automatically generate a caption (comprising a sequence of words) for the digital content item.” [0096] “the appearance and geometry features are used by the ORT's encoder to generate encoded output, as is discussed in more detail below. At step 410 and as is discussed in more detail below, the encoded output is used by the ORT's decoder to automatically generate a caption corresponding to the digital content item. At step 412, the digital content item and the automatically-generated caption are communicated to the user. By way of one non-limiting example, the digital content item and automatically-generated caption can be communicated by a server (e.g., content server 106, application server 108, search server 120, etc.) over an electronic communications network (e.g., the Internet) to an application (e.g., browser application) executing on the user's computing device (e.g., client device, mobile device 200, etc.) resulting in the digital content item being displayed with the automatically-generated caption in the user interface (e.g., an application user interface) of the user's computing device. In addition or as an alternative to visual presentation of the automatically-generated caption, the automatically-generated caption can be transmitted (e.g., via the Internet) to a user device causing the automatically-generated caption to be output in an audible format—e.g., text of the automatically-generated caption can be converted to audio output using text-to-speech software—at the user device (e.g., so that the user, such as a visually-impaired user, can listen to the caption to discern the image's contents).” [0105] “ORT 610 is a transformer model, or transformer, which is a type of neural network architecture, which is trained to “translate” a set of objects detected in a digital content item to generate a caption comprising a sequence of words. In accordance with one or more embodiments, the “translation” considers determined spatial relationships between the objects detected in the digital content item. In accordance with one or more embodiments, the transformer (e.g., ORT 610) architecture can comprise one or more neural networks (e.g., convolutional neural network) together with a number of attention models. The transformer's neural network architecture (including learned functions, weights, etc.) can be trained using machine learning together with one or more datasets used to train as well as test and evaluate the trained model.” [0148] “Persistent storage, e.g., medium/media 1206, can be used to store an operating system and one or more application programs. Persistent storage can also be used to store device drivers, such as one or more of a digital camera driver, monitor driver, printer driver, scanner driver, or other device drivers, web pages, content files, playlists and other files.” [0152]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Herdade into Messer, in order to provide a convenient mechanism for presenting a textual description of the reproduced scene to a user.
The combination of Messer and Herdade does not explicitly teach perform reproduction processing on a scene shown in the camera footage, wherein the reproduction processing comprises; rendering an abstracted image of a space included in the camera footage based on the recognition information on a static object included in the scene information; rendering an abstracted image of the human included in the camera footage on the abstracted image of the space based on the recognition information on the human included in the scene information; and generating a reproduced image by adding the information to the abstracted image of the space in which the abstracted image of the human is rendered. This is what Bolle teaches. Bolle teaches perform reproduction processing on a scene shown in the camera footage, (“A video source 310 represents one or more video sources. Each video source could be either be a live camera or video file representing a camera input. Further, the camera could be either a static camera or dynamically controlled (e.g., pan-tilt-zoom) camera. The video consists of a sequences of images captured at successive time instants.” [0060]“Given the (private) keys released by the authorization module 830, decryption method 820 (FIG. 9) takes an encrypted (e.g., encoded by public keys as in DES) version of the track attributes, encrypted background and encrypted raw video, generates a subset of transformed attributes of the tracks, background and raw video. What subset of the transformed track attributes are generated depends upon depending upon the number of valid (private) keys received. … Given (a subset of) transformed attributes of the tracks, background and raw video, synthesis method (870, FIG. 8) generates the synthesized transformed video. Depending upon the authorization of the users, they may be able to directly access the video or will be able only access the statistical information in the transformed video through statistical query processor 490. … the decoding system obtain the next encrypted overall encrypted information (757). It starts with initializing resetting a fresh decoded frame (803). The initialized state of the decoded frame of video may be prescribed by the system policies at the video decoding end, or they may be determined by the outcome of the decoding process of the last frame. … it initializes the parameters ofthe video reconstruction (e.g., parameters of the 3D model). Again the default video parameters may be prescribed by the system policies or by the system inertia retained from the last frame. It then fetches the overall (private) key from the key generation process (807). In the next step (809), the system uses the overall key to decrypt the encrypted overall encrypted information of the current frame. If the user is already authenticated (by a system determined distinctive possession, knowledge, or biometrics) and is authorized to access the system, the overall key obtained from the key generation process will be able to decrypt the encrypted overall encrypted information. Based on the output of the decryption process, a test is performed to determine the success of the decryption process (811). If the test is not successful, the system determines overall authorization failure and the situation is handled as determined by the system policy. If the overall decryption is successful, the decoding system now has access to overall vector of encrypted information (M, 755, FIG. 7A). It then obtains other keys K. specific to the information content X the user is authorized to access (813) from the key generation process (830). In step 815, these specific keys K. are then used to decrypt the corresponding specific information content in the contained in overall vector of encrypted information (M, 755, FIG. 7A). Note that access to specific key K. does not necessarily mean access to information content X as the information X may have been blocked out at the encoding end. In step 817, the decrypted video information is passed on to the transformed video synthesis process (870, FIG. 8) and the decoded raw video process (860, FIG. 8).” [0169-0172]) Bolle also teaches the reproduction processing comprises; rendering an abstracted image of a space included in the camera footage based on the recognition information on a static object included in the scene information; (“A video source 310 represents one or more video sources. Each video source could be either be a live camera or video file representing a camera input. Further, the camera could be either a static camera or dynamically controlled (e.g., pan-tilt-zoom) camera. The video consists of a sequences of images captured at successive time instants. … A background estimation module 320 is designed to estimate the information about the appearance relatively static objects in the scene, e.g., furniture, walls, sky etc. One method for background estimation assumes that the background is constant and will never change. This method will estimate the background objects only once (e.g., by using one or more video frames of the scene when no objects are in the field of view of the camera pointing to the scene) and from there onwards, use these static estimates of the background for the rest of the video processing.” [0060-0061] “The background model as estimated by 520 can be completely removed from the scene to avoid dissemination of any information about the location of the scene. … The background model as estimated by 520 can be replaced by a general symbolic description to avoid dissemination of detailed information about the location of the scene. … The background model as estimated by 520 can be replaced by sufficiently general (perhaps, distorted) sketch of the scene to avoid dissemination of any detailed information about the location of the scene. … The background model as estimated by 520 can be replaced by a different background to avoid dissemination of information about the location of the scene. One method of location obscuration 603 is by background model estimate rendered with a different resolution and/or with pseudo-color texture. For example, the background model as estimated by 520 can be replaced by a blurred version of the background estimate to avoid dissemination of detailed information (e.g., number of windows) about the location of the scene. … The random displacement of the pixels will create a meaningless background texture but the statistical analysis 890 may be able to infer some overall properties of the background (e.g., color of the background, is it likely to be outdoor) without actually revealing any particular information about the background location.” [0127-0132]) Bolle further teaches rendering an abstracted image of the human included in the camera footage on the abstracted image of the space based on the recognition information on the human included in the scene information; (“A video source 310 represents one or more video sources. Each video source could be either be a live camera or video file representing a camera input. Further, the camera could be either a static camera or dynamically controlled (e.g., pan-tilt-zoom) camera. The video consists of a sequences of images captured at successive time instants.” [0060] “A part identification module 370 decomposes the foreground regions, classified by the tracking module 350 (say, as humans), into regions that represent parts of object (e.g., parts of human body). … Once the body parts are identified, an estimate of the location, size, and pose of the body part can be estimated. … The determination of position, size, shapes of parts of a human body as estimated by the part identification module 370 at different video frames, i.e., times enables a part tracking module 380 to compute/estimate different additional attributes of the parts of the body (e.g., average speed, average velocity, instantaneous speed distributions, 3D orientation) related to the trajectory of the body parts. … Given the body part position, size, and pose for each part at different times, the part tracking module 380 determines one or more trajectory based attributes of (e.g., velocity, speed, orientations of human body parts at different video frames, i.e., times) parts of a human body. … Given actions of the individuals in the scene and their body part positions, sizes, and poses at different times, the interactions (e.g., hand shake, fight) between the individuals in the scene can be concluded by the part tracking module 380. Similarly, actions of the individuals and their interrelationship with other objects reveal information about object-human interactions.” [0091-0095] “Each track encoding (object pose, overall object region silhouette, overall object appearance, overall object movement, object part movements, object actions, object interactions, object category, object identity) and background encoding (representation describing background scene) contains sufficient information in them to synthesize the object up to the information contained in the video to varying levels of abstraction representation.” [0101]) Bolle teaches generating a reproduced image by adding the information to the abstracted image of the space in which the abstracted image of the human is rendered. (“A preferred instantiation of the video analysis system 120 results in information about the background (static) objects in the scene and about the moving objects (e.g., foreground) objects in the scene. A more preferred instantiation of 120 presents many additional attributes (e.g., locations, category, actions, interactions) of the foreground objects in the scene at successive times (e.g., frames). Further, 120 classifies each foreground object into various categories, at least one category being humans. Examples of other categories include vehicles, animals.” [0037] “A part identification module 370 decomposes the foreground regions, classified by the tracking module 350 (say, as humans), into regions that represent parts of object (e.g., parts of human body). This decomposition may be based on the shape of the silhouette of the human body, the image texture (appearance), motion information (e.g., portions of the foreground region depicting a human body, the current video frame most similar to the portions of the corresponding foreground region in the previous video depicting a known body part). This functionality is often termed as shape (body part) identification.” [0091] “The extent/nature of recovery of original (210, 821) information depends upon appropriate authorizations released from authentication module 142. The extent of the recovery of the information from encoded video depends upon the nature of keys/authorizations released by 830 which in their turn depend upon the decisions of authentication module 142. In other words, the loss of information content from original video (110,210), original transformed video parameters as generated in 230 to decoded raw video 860, decoded transformed video parameters 821 depends upon the key generation/authorization 830 released by the user authentication procedures 142. Given the recovered parameters of the transformed video 821, 870 synthesizes the transformed video 871. … The decoding of the video itself does not necessarily imply user access to the results. The system permit direct user access to the selectively synthesized transformed video (897) and/or selectively decoded raw video (896) depending upon the sets of authorizations released by the user authentication module 142. … Security personnel (after successful completion of appropriate authentication procedures 142) on the premise will be authorized (875) to see the transformed video (897 which is synthesized from part or full of the parameters of the transformed video information in 330) but may not be able to see the original video. … Given (a subset of) transformed attributes of the tracks, background and raw video, synthesis method (870, FIG. 8) generates the synthesized transformed video.” [0166-0170]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Bolle into the combination of Messer and Herdade, in order to provide a semantically simplified, privacy-preserving and more informative reproductions of recorded surveillance footage.
9. With reference to claim 2, Messer teaches the processing circuitry is configured to: when the recognition information on the object includes recognition information on a moving object other than a human, add recognition information on the moving object to the scene information, (“The camera apparatus may comprise an object recognition module for classifying objects in captured images and/or video clips. Preferably the object recognition module is arranged to process captured image data/video clip data which corresponds to a captured image/captured video clip in which an object of interest has been first detected by the object detection module, to thereby classify objects of interest.” [0026] “The camera apparatus may be arranged to allow the capture of images and/or video clips from recorded video, say as this is playback at the camera apparatus. Such captured images and/or video clips may then be treated/processed as described above in the same way as images and/or video clips captured when live video is being captured and corresponding object recognition data relating to such captured images and/or video clips may be sent along with the recorded video where this is sent via the video encoder. The server may be arranged to send packages of data to end user devices for allowing end users to view video received from camera apparatus and/or for allowing end users to be informed of object recognition events. The server may be arranged to add meta data which relates to an object that has been recognised to at least some such packages of data. The meta data may include GPS data, image/video analytics data, unique ID data, location data indicating a region of interest in an image—eg of a detected face, time-stamp data, device data, etc.” [0084-0086] “The objects of interest can in principle be anything which it is desired to monitor, detect and recognise. Specific examples include faces—ie people, vehicle registration numbers, weapons.” [0093])
The combination of Messer and Herdade does not explicitly teach the reproduction processing further comprises: rendering an abstracted image of the moving object on the abstracted image of the space based on the recognition information on the moving object included in the scene information. This is what Bolle teaches (“A preferred instantiation of the video analysis system 120 results in information about the background (static) objects in the scene and about the moving objects (e.g., foreground) objects in the scene. A more preferred instantiation of 120 presents many additional attributes (e.g., locations, category, actions, interactions) of the foreground objects in the scene at successive times (e.g., frames). Further, 120 classifies each foreground object into various categories, at least one category being humans. Examples of other categories include vehicles, animals.” [0037] “A tracking module, 350 relates the foreground regions of frame n to those from earlier frames. Each foreground region in the current frame may represent an entirely new object, may correspond to foreground region from previous frames depicting moving object. Some of the previously seen objects (e.g., foreground regions from previous video frames) do not appear in the current frame because either these objects may have been occluded by other objects or these objects may have left the field of the camera. Locations of foreground regions corresponding to a single object in the successive video frames defines a track. The process of identifying the relationship among the foreground regions of different video frames (e.g., do two given foreground regions from two video frames depict same object or different object) is called tracking.” [0077] “The extent/nature of recovery of original (210, 821) information depends upon appropriate authorizations released from authentication module 142. The extent of the recovery of the information from encoded video depends upon the nature of keys/authorizations released by 830 which in their turn depend upon the decisions of authentication module 142. In other words, the loss of information content from original video (110,210), original transformed video parameters as generated in 230 to decoded raw video 860, decoded transformed video parameters 821 depends upon the key generation/authorization 830 released by the user authentication procedures 142. Given the recovered parameters of the transformed video 821, 870 synthesizes the transformed video 871. … The decoding of the video itself does not necessarily imply user access to the results. The system permit direct user access to the selectively synthesized transformed video (897) and/or selectively decoded raw video (896) depending upon the sets of authorizations released by the user authentication module 142. … Security personnel (after successful completion of appropriate authentication procedures 142) on the premise will be authorized (875) to see the transformed video (897 which is synthesized from part or full of the parameters of the transformed video information in 330) but may not be able to see the original video. … Given (a subset of) transformed attributes of the tracks, background and raw video, synthesis method (870, FIG. 8) generates the synthesized transformed video.” [0166-0170]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Bolle into the combination of Messer and Herdade, in order to provide a semantically simplified, privacy-preserving and more informative reproductions of recorded surveillance footage.
10. With reference to claim 5, The combination of Messer and Herdade does not explicitly teach the abstracted image of the space includes a three-dimensional image obtained by abstracting the space shown in the camera footage; and the abstracted image of the human includes a three-dimensional image obtained by abstracting the human shown in the camera footage. This is what Bolle teaches (“A video source 310 represents one or more video sources. Each video source could be either be a live camera or video file representing a camera input. Further, the camera could be either a static camera or dynamically controlled (e.g., pan-tilt-zoom) camera. The video consists of a sequences of images captured at successive time instants.” [0060] “A part identification module 370 decomposes the foreground regions, classified by the tracking module 350 (say, as humans), into regions that represent parts of object (e.g., parts of human body). … Once the body parts are identified, an estimate of the location, size, and pose of the body part can be estimated. … The determination of position, size, shapes of parts of a human body as estimated by the part identification module 370 at different video frames, i.e., times enables a part tracking module 380 to compute/estimate different additional attributes of the parts of the body (e.g., average speed, average velocity, instantaneous speed distributions, 3D orientation) related to the trajectory of the body parts. … Given the body part position, size, and pose for each part at different times, the part tracking module 380 determines one or more trajectory based attributes of (e.g., velocity, speed, orientations of human body parts at different video frames, i.e., times) parts of a human body. … Given actions of the individuals in the scene and their body part positions, sizes, and poses at different times, the interactions (e.g., hand shake, fight) between the individuals in the scene can be concluded by the part tracking module 380. Similarly, actions of the individuals and their interrelationship with other objects reveal information about object-human interactions.” [0091-0095] “Each track encoding (object pose, overall object region silhouette, overall object appearance, overall object movement, object part movements, object actions, object interactions, object category, object identity) and background encoding (representation describing background scene) contains sufficient information in them to synthesize the object up to the information contained in the video to varying levels of abstraction representation.” [0101] “it initializes the parameters of the video reconstruction (e.g., parameters of the 3D model). Again the default video parameters may be prescribed by the system policies or by the system inertia retained from the last frame. It then fetches the overall (private) key from the key generation process (807).” [0172]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Bolle into the combination of Messer and Herdade, in order to provide a semantically simplified, privacy-preserving and more informative reproductions of recorded surveillance footage.
11. Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Messer et al. (US 2022/0020255 A1) in view of Herdade et al. (US 2021/0201044 A1) and Bolle et al. (US 2007/0201694 A1), as applied to claim 1 above, and further in view of Hung (US 2022/0343893 A1).
12. With reference to claim 3, the combination of Messer, Herdade and Bolle does not explicitly teach the processing circuitry is further configured to: detect a specific language set in advance by referring to the linguistic information on the scene included in the scene information; when the linguistic information on the scene including the information on the specific language is detected, regenerates detailed linguistic information on the scene shown in the camera footage by performing the linguistic processing again on the camera footage that is a generation source of the detected linguistic information on the scene; and update the scene information including the information on the specific language based on the detailed linguistic information on the scene. This is what Hung teaches (“The data retrieval engine 151 accesses electronic content 140 comprising one or more types of audio-visual data including video data, image data, holographic data, 3-D image data, etc., wherein the data retrieval engine 151 is able to extract or retrieve audio streams 142 from the electronic content 141. … The data retrieval engine 151 locates, selects, and/or stores raw recorded source data wherein the data retrieval engine 151 is in communication with one or more other ML engine(s) and/or models included in computing system 110.” [0033-0034] “The electronic content 141 can be obtained in real-time from an audio-visual or audio streaming device/application or can be obtained from previously recorded data (e.g., from remote systems 120).” [0043] “The first speech recognition engine is also selectable based on user-defined settings, such as pre-defining a subset of languages or a user language preference.” [0051] “Subsequent to and/or concurrently to generating the second transcription, the LID module 152A is used to determine that the second portion of the audio stream or that the temporal location T2 of the audio stream corresponds to a second language (e.g., Language B 148B). … Subsequent to determining that the second portion (e.g., Audio Portion 206) of the audio stream 142A corresponds to a second language (e.g., Language B 148B), the system (e.g., computing system 110) generates a third transcription which is a corrected transcription (e.g., Transcript BT2 (B) 145A) from the Audio Portion 206 stored in the internal buffer 147. … After the corrected transcription is generated, the system replaces and/or overwrites (e.g., overwrite arrow 212) the second transcription (e.g., Transcript AT2 (RT) 144B stored in the transcription cache 147) with the third transcription (e.g., Transcript BT2 (B) 145A) in the transcription cache 147.” [0056-0058] “the transcriptions output by SR (en-US) 402 at T2 and T3 are incorrect and need to be overwritten. The system then retrieves audio portions at T2 and T3 from the internal buffer (e.g., internal buffer 147 of FIGS. 1 and 2) and transmits them to SR (zh-CN) 406, which then generates a corrected transcript. The system is configured to go back to T2, for example, or another earlier portion in the audio stream when the first errors or non-matching phonemes for a previous language were detected.” [0071] “Subsequent to selecting the second or other subsequent speech recognition engine and retrieving the buffered portion of the audio stream, the system generates a new transcription (e.g., processed results 514) of the buffered portion 512 of the audio stream in the second language using the second speech recognition engine SR #2 510. The incorrect recognition results (which were processed by the first speech recognition engine before the handoff was completed) are removed from the transcription cache and replaced with the correct results. Additionally, if results are being displayed and previous transcription results were displayed from the first speech recognition engine that are incorrect, the system replaces the incorrect results on the display interface with new corrected results generated from the second speech recognition engine.” [0079-0080] “A language identification module configured to continuously analyze the audio stream and detect languages corresponding to spoken language utterances is also selected by the system automatically based on predetermined settings, or profiles or contexts of the audio stream and/or based on user input (act 930). Subsequent to selecting the language identification module and obtaining the audio stream, the language identification modules is used to start analyzing the audio stream starting at the first temporal location (act 935). While generating the real-time transcription of the audio stream in the first language, language identification module is used to detect that a new spoken utterance starting at a subsequent temporal location in the audio stream corresponds to a second language (act 940). Subsequent to detecting that the new spoken utterance in the audio stream corresponds to the second language, the computing system selects a second speech recognition engine configured to transcribe spoken language utterances in the second language (act 945) and retrieves a buffered portion of the audio stream from the internal buffer, the buffered portion starting at the subsequent temporal location (act 950). … the system may effectively and dynamically select and switch the speech recognition engines that are used to transcribe the different languages that are detected, dynamically switching between the different speech recognition engines that are used for each of the different corresponding detected language switches in the audio stream, also replacing any portions of incorrectly transcribed audio segments associated with the different switches that may have been incorrectly transcribed before and/or during the processes of making the switch(es) between the different speech recognition engine(s).” [0097-0102]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Hung into the combination of Messer, Herdade and Bolle, in order to provide more accurate/detailed linguistic information associated with the recorded scene.
13. Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Messer et al. (US 2022/0020255 A1) in view of Herdade et al. (US 2021/0201044 A1) and Bolle et al. (US 2007/0201694 A1), as applied to claim 1 above, and further in view of Hsieh et al. (US 2020/0327160 A1).
14. With reference to claim 4, the combination of Messer, Herdade and Bolle does not explicitly teach an input device to which information is input, wherein the processing circuitry is further configured to: when search information on a scene shown in the camera footage is input from the input device, identify the scene information matching the search information based on the search information and the linguistic information on the scene included in the scene information stored in the memory device, wherein the reproduction processing is performed based on the scene information matching the search information. This is what Hsieh teaches (“the generation component 140 may generate frame keywords using a language model. The language model may be part of or associated with the image model, such as embodiments where the image model is at least a part of the recognition component 130. The language model may translate features, objects, aspects, nuances, subjects, or other characteristics, identified by the image model, into natural language segments. The natural language segments of the language model may include frame keywords, sentences, or other recognizable language segments.” [0032] “generating the index store includes generating a time series frame index. The time series frame index may be generated with keywords associated with movement.” [0037] “the index component 160 incorporates text for the image captions for the video content into the index store. Once the natural language phrases have been generated by the generation component 140, the index component 160 may incorporate the text of the image captions into the index store. As the video content is replayed, after inserting the text into the index store, the text may be portrayed on a display device.” [0041] “the interface component 170 receives a search query within a user interface. The search query may contain one or more search terms. The search query may be received within a text entry field, one or more drop down menus, a touch screen interface, a voice command, combinations thereof, or any other suitable input mechanism, method, or interface. The search query may be a natural language query, a Boolean query, proximity query, truncation query, wildcard query, combinations thereof, or any other suitable search query type. The search terms may be selected from the search query by identifying Boolean or other operators indicating relationships between the search terms. In some embodiments, the interface component 170 or the search component 180 identifies the search terms, and interrelations between the search terms, of a natural language query using one or more natural language processing operations.” [0043] “he search component 180 may run a plurality of searches, with each search associated with an identified search type. The plurality of searches may be understood as partial searches producing partial search results. The partial search results from the plurality of searches may then be aggregated, appended, interwoven, pruned, combinations thereof, or otherwise combined to produce full search results. In some embodiments, the search component 180 determines an existence of a plurality of search types and determines a relative importance of the search types. For example, the search component 180 may receive a search query of “A woman in a red shirt hugs a koala.” The search component 180 may determine a first search type (e.g., primary search type) as a movement search based, at least in part, on the verb “hug.”” [0045] “the search component 180 compares the at least one search term with one or more frame keywords within the time series frame index. Where the at least one search term and the one or more frame keywords or other descriptive elements are a match or are sufficiently related, the search component 180 may return frames identified by the time series frame index as a potential result. In embodiments where the search component 180 identifies a plurality of time series frame indexes, the search component 180 may pare the plurality of time series frame indexes into a subset of time series frame indexes. The subset of time series frame indexes may be one or more time series frame indexes having descriptors or frame keywords with a closest match to the at least one search term.” [0052] “the operations further comprise: receiving a search query, within a user interface, containing one or more search terms; determining a search type associated with each search term within the search query; searching the video content based on the search type associated with the search terms within the search query; and generating joint search results based on the search type and the search terms of the search query and the frame keywords and the time frames of the index store.” claim 11 “the search type for at least one search term is associated with movement and searching the video content based on the search type further comprises: identifying a time series frame index for the video content; comparing the at least one search term with one or more frame keywords within the time series frame index; and generating a partial search result indicating at least one time frame within the video content.” claim 12) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Hsieh into the combination of Messer, Herdade and Bolle, in order to improve retrieval/searchability of stored camera footage.
Conclusion
15. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michelle Chin whose telephone number is (571)270-3697. The examiner can normally be reached on Monday-Friday 8:00 AM-4:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http:/Awww.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Kent Chang can be reached on (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https:/Awww.uspto.gov/patents/apply/patent- center for more information about Patent Center and https:/Awww.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE CHIN/
Primary Examiner, Art Unit 2614
Prosecution Insights