What is log, pre-processing?

Log pre-processing is a critical phase in log management and Security Information and Event Management (SIEM) where raw log data is prepared for analysis. The most fundamental step in this phase is normalization, which involves parsing and converting log entries from their many different native source formats into a single, standardized, or common format. This conversion is essential for the SIEM to effectively correlate and analyze events from disparate systems, such as firewalls, operating systems, and applications. Without a common format, comparing and understanding the relationship between events from different sources would be impossible.

A. Removing known bad log event entries is a form of filtering, which is one possible action during pre-processing, but not its core definition.

C. Moving log entries to an analyst's queue is part of the event triage and incident response workflow, which occurs after logs are processed and correlated.

D. Transferring logs to short-term storage is a function of log collection and aggregation, not the processing of the log data's content and format.

---

1. National Institute of Standards and Technology (NIST). (2006). Guide to Computer Security Log Management (Special Publication 800-92).

Section 3.2

Log Parsing

Page 3-3: "Log parsers convert log entries from their native format into a common format. Because each log source may have its own log format

a log parser needs to be created for each unique log format. Once logs have been converted to a common format

they can be processed by other applications

such as log viewers and analysis tools." This directly supports the process of converting formats.

2. Dain

O.

& Cunningham

R. K. (2001). Fusing a Heterogeneous Alert Stream into a Coherent Attack Scenario. Proceedings of the 2001 ACM workshop on Intrusion detection systems.

Section 3.1

Normalization: "The first step in the fusion process is to translate the heterogeneous alert formats into a common frame of reference... The normalization process converts each alert into our internal representation." This academic paper on intrusion detection highlights normalization (conversion) as the initial

essential step.

DOI: https://doi.org/10.1145/501991.502001

3. Stanford University. (2021). CS 155: Computer and Network Security

Lecture 16: Logging and Auditing.

Slide 19

"Log Analysis": The lecture notes discuss the challenge of analyzing logs from diverse sources

stating the need to "Parse unstructured logs into a structured format" and "Normalize fields into a canonical format." This university courseware identifies parsing and normalization (conversion) as a key step for log analysis.

4. Elastic. (n.d.). Logstash Introduction. Official Elastic Documentation.

"What is Logstash?" section: "Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources

transforms it

and then sends it to your favorite 'stash.'... Because of its tight integration with Beats and Elasticsearch

Logstash is the tool of choice for loading data into Elasticsearch." The "transform" function described here is the pre-processing step of converting data formats.