Applied security visualization

APPLIED SECURITY VISUALIZATION

"Collecting log data is one thing, having relevant information is something else. The art to transform all kinds of log data into meaningful security information is the core of this book. Raffy illustrates in a straight forward way, and with hands-on examples, how such a challenge can be mastered. Let''s get inspired."

-Andreas Wuchner, Head of Global IT Security, Novartis

Use Visualization to Secure Your Network Against the Toughest, Best-Hidden Threats

As networks become ever more complex, securing them becomes more and more difficult. The solution is visualization. Using today''s state-of-the-art data visualization techniques, you can gain a far deeper understanding of what''s happening on your network right now. You can uncover hidden patterns of data, identify emerging vulnerabilities and attacks, and respond decisively with countermeasures that are far more likely to succeed than conventional methods.

In Applied Security Visualization , leading network security visualization expert Raffael Marty introduces all the concepts, techniques, and tools you need to use visualization on your network. You''ll learn how to identify and utilize the right data sources, then transform your data into visuals that reveal what you really need to know. Next, Marty shows how to use visualization to perform broad network security analyses, assess specific threats, and even improve business compliance.

He concludes with an introduction to a broad set of visualization tools. The book''s CD also includes DAVIX, a compilation of freely available tools for security visualization.

You''ll learn how to:

* Intimately understand the data sources that are essential for effective visualization

* Choose the most appropriate graphs and techniques for your IT data

* Transform complex data into crystal-clear visual representations

* Iterate your graphs to deliver even better insight for taking action

* Assess threats to your network perimeter, as well as threats imposed by insiders

* Use visualization to manage risks and compliance mandates more successfully

* Visually audit both the technical and organizational aspects of information and network security

* Compare and master today''s most useful tools for security visualization

Contains the live CD Data Analysis and Visualization Linux (DAVIX). DAVIX is a compilation of powerful tools for visualizing networks and assessing their security. DAVIX runs directly from the CD-ROM, without installation.

Raffael Marty is chief security strategist and senior product manager for Splunk, the leading provider of large-scale, high-speed indexing and search technology for IT infrastructures. As customer advocate and guardian, he focuses on using his skills in data visualization, log management, intrusion detection, and compliance. An active participant on industry standards committees such as CEE (Common Event Expression) and OVAL (Open Vulnerability and Assessment Language), Marty created the Thor and AfterGlow automation tools, and founded the security visualization portal secviz.org. Before joining Splunk, he managed the solutions team at ArcSight, served as IT security consultant for PriceWaterhouseCoopers, and was a member of the IBM Research Global Security Analysis Lab.

• Preface p. xiii
• Acknowledgments p. xix
• About the Author p. xxiii
• Chapter 1 Visualization p. 1
• What Is Visualization? p. 2
• Why Visualization? p. 3
• Visualization Benefits p. 5
• Security Visualization p. 6
• Security Visualization's Dichotomy p. 7
• Visualization Theory p. 8
• Perception p. 9
• Expressive and Effective Graphs p. 11
• Graph Design Principles p. 13
• Information Seeking Mantra p. 18
• Summary p. 19
• Chapter 2 Data Sources p. 21
• Terminology p. 22
• Security Data p. 23
• Common Problems p. 24
• Incomplete Information p. 25
• Source/Destination Confusion p. 26
• Packet Captures p. 27
• Traffic Flows p. 30
• Collecting Traffic Flows p. 32
• Aggregating Traffic Flows p. 35
• Clustering Traffic Flows p. 36
• Anonymizing Traffic Flows p. 36
• Firewalls p. 37
• Intrusion Detection and Prevention Systems p. 40
• Passive Network Analysis p. 43
• Operating Systems p. 45
• Real-Time Operating System Information p. 46
• Operating System State Information p. 49
• Operating System Log Problems p. 53
• Applications p. 55
• Web Proxy p. 56
• Mail p. 58
• Databases p. 60
• Configurations p. 62
• Summary p. 64
• Chapter 3 VisuallyRepresenting Data p. 65
• Graph Properties p. 66
• Data Types p. 66
• Color p. 68
• Size, Shape, and Orientation p. 69
• Chart Axes p. 69
• Simple Charts p. 70
• Pie Chart p. 71
• Bar Chart p. 72
• Line Chart p. 73
• 3D Bar Charts p. 74
• Stacked Charts p. 75
• Stacked Pie Chart p. 76
• Stacked Bar Chart p. 77
• Stacked Line Chart p. 78
• Histograms p. 78
• Box Plots p. 80
• Scatter Plots p. 82
• Parallel Coordinates p. 85
• Link Graphs p. 87
• Maps p. 93
• Treemaps p. 96
• Three-Dimensional Views p. 100
• Three-Dimensional Scatter Plots p. 101
• Three-Dimensional Link Graphs p. 103
• Interaction and Animation p. 104
• Interaction p. 104
• Animation p. 105
• Choosing the Right Graph p. 109
• Challenges p. 115
• Summary p. 117
• Chapter 4 From Data to Graphs p. 119
• Information Visualization Process p. 119
• Step 1 Define the Problem p. 121
• Step 2 Assess Available Data p. 122
• Step 3 Process Information p. 124
• Adding Additional Data p. 126
• Filtering Log Entries p. 127
• Aggregation p. 128
• Data Processing Challenges p. 129
• Step 4 Visual Transformation p. 132
• Data Mapping p. 132
• Size and Shape p. 137
• Color p. 140
• Step 5 View Transformation p. 143
• Aggregation p. 144
• Step 6 Interpret and Decide p. 146
• Tools for Data Processing p. 150
• Excel, OpenOffice, and Text Editors p. 151
• Regular Expressions p. 151
• UNIX tools p. 152
• Perl p. 155
• Parsers p. 157
• Other Tools p. 158
• Summary p. 158
• Chapter 5 Visual Security Analysis p. 161
• Reporting p. 162
• Reporting Tools p. 164
• Issues and Problems p. 165
• Reporting Machine Access-An Example p. 165
• Historical Analysis p. 169
• Time-Series Visualization p. 169
• Correlation Graphs p. 189
• Interactive Analysis p. 192
• Forensic Analysis p. 197
• Real-Time Monitoring and Analysis p. 228
• Dashboards p. 228
• Situational Awareness p. 236
• Summary p. 237
• Chapter 6 Perimeter Threat p. 239
• Traffic-Flow Monitoring and Analysis p. 240
• Service Characteristics p. 240
• Service Anomalies p. 245
• Worm Detection p. 250
• Denial of Service p. 254
• Botnets p. 257
• Policy-Based Traffic-Flow Analysis p. 264
• Firewall Log Analysis p. 268
• Firewall Visualization Process p. 268
• Firewall Ruleset Analysis p. 272
• Intrusion Detection System Signature Tuning p. 278
• Wireless Sniffing p. 286
• Email Data Analysis p. 290
• Email Server Analysis p. 291
• Social Network Analysis p. 298
• Vulnerability Data Visualization p. 302
• Risk-Posture Visualization p. 304
• Vulnerability-Posture Changes p. 310
• Summary p. 312
• Chapter 7 Compliance p. 315
• Policies, Objectives, and Controls p. 316
• Regulations and Industry Mandates p. 318
• IT Control Frameworks p. 322
• Logging Requirements p. 324
• Audit p. 328
• Audit Data Visualization p. 332
• Business Process Monitoring p. 333
• Compliance Monitoring p. 338
• Risk Management p. 343
• Control Objective Prioritization p. 345
• Risk Visualization p. 346
• Separation of Duties p. 356
• An Example of Applying Visualization to an SoD Audit p. 357
• Generating SoD Graphs p. 360
• Database Monitoring p. 362
• Summary p. 370
• Chapter 8 Insider Threat p. 373
• Insider Threat Visualization p. 374
• What Is a Malicious Insider? p. 374
• Three Types of Insider Crimes p. 375
• Information Theft p. 376
• Fraud p. 382
• Sabotage p. 387
• Who Are the Malicious Insiders? p. 390
• Information Theft p. 390
• Fraudster p. 391
• Saboteur p. 391
• A Detection Framework for Malicious Insiders p. 392
• Precursors p. 392
• Assigning Scores to Precursors p. 394
• Insider-Detection Process p. 396
• Summary of Insider-Detection Process p. 408
• Insider-Detection Process at Work p. 409
• Improved Insider-Detection Process p. 414
• Watch Lists p. 415
• Adding Watch Lists to the Insider-Detection Process p. 419
• Grouping Precursors into Buckets p. 420
• Candidate Graph Based on Precursor Buckets p. 422
• Improved Insider-Detection Process Summary p. 424
• Extended Insider-Detection Process at Work p. 424
• Challenges p. 431
• Proactive Mitigation p. 432
• Sample Precursors p. 433
• Summary p. 444
• Chapter 9 Data Visualization Tools p. 445
• Data Inputs p. 446
• Comma Separated Values p. 446
• TM3 p. 447
• DOT p. 448
• GML p. 449
• Freely Available Visualization Tools p. 450
• Static Data Graphs p. 451
• Stand-Alone Applications p. 464
• Open Source Visualization Libraries p. 492
• Java Libraries p. 493
• Non-Java Libraries p. 494
• Charting Libraries p. 495
• Libraries Summary p. 496
• Online Tools p. 497
• Swivel p. 498
• Many Eyes p. 499
• Google Maps and Google Earth p. 499
• Google Chart API p. 501
• Commercial Visualization Tools p. 502
• Advizor p. 502
• Other Commercial Visualization Tools p. 504
• Summary p. 505
• Index p. 507

Preface This book is about visualizing computer security data. The book shows you, step by step, how to visually analyze electronically generated security data. IT data must be gathered and analyzed for myriad reasons, including GRC (governance, risk, and compliance) and preventing/mitigating insider threats and perimeter threats. Log files, configuration files, and other IT security data must be analyzed and monitored to address a variety of use-cases. In contrast to handling textual data, visualization offers a new, more effective, and simpler approach to analyzing millions of log entries generated on a daily basis. Graphical representations help you immediately identify outliers, detect malicious activity, uncover misconfigurations and anomalies, and spot general trends and relationships among individual data points. Visualization of data--the process of converting security data into a picture--is the single most effective tool to address these tasks. After all... A picture is worth a thousand log entries. To handle today's security and threat landscape, we need new analysis methods. Criminal activity is moving up the network stack. Network-based attacks are becoming more sophisticated, and increasingly attacks are executed on the application layer. Criminal techniques have adapted. Are you prepared to deal with these new developments? Are you aware of what is happening inside of your networks and applications? In addition to monitoring your networks, you must make sure you are taking an in-depth look at your applications. Because of the vast amount of data that requires analysis, novel methods are needed to conduct the analysis. Visualization can help address these complex data analysis problems. What This Book Covers Follow me on an exciting journey through security data visualization. We will start with the basics of data sources needed for security visualization. What are they? What information do they contain, and what are the problems associated with them? I then discuss different ways to display data in charts or more complex visualizations, such as parallel coordinates. You will learn which graphical methods to use and when. The book then takes you through the process of generating graphical representations of your data. A step-by-step approach guarantees that no detail is left out. By introducing an information visualization process, visualization of security data becomes a simple recipe, which I apply in the core of this book to analyze three big areas of security visualization: perimeter threat, compliance, and insider threat. These chapters are hands-on and use-case driven. Open source visualization tools and libraries are discussed in the last chapter of the book. You can find all the tools introduced on the accompanying CD. Without dealing with installations, you can immediately start analyzing your own security data. The book is a hands-on guide to visualization. Where it covers theoretical concepts and processes, it backs them up with examples of how to apply the theory on your own data. In addition to discussing--step by step--how to generate graphical representations of security data, this book also shows you how to analyze and interpret them. The goal is to get you excited and inspired. You are given the necessary tools and information to go ahead and embed visualization in your own daily job. The book shows example use-cases that should inspire you to go ahead and apply visualization to your own problems. If one of the chapters covers a topic that is not your responsibility or focus area (for example, compliance), try to see beyond the topic specifics and instead explore the visualizations. The concepts may be valid for other use-cases that you want to address. What This Book Doesn't Cover - This book covers visualization of computer security data. I do not discuss topics such as binary code or malware analysis. I don't get into the topics of steganography (the art or science of hiding information in images) or system call visualizations. This book is about time-based data and system status records. The data visualized is data you use to operationally secure an organization. This book is not a compendium of security data sources and possible visual representations. It uses existing visualization methods--charts, parallel coordinates, treemaps, and so on--that are supported by many tools and applications. The book is composed of a sample set of data sources and use-cases to illustrate how visualization can be used. Audience I wrote this book for security practitioners. I am introducing new ways to analyze security data to the people who can implement them. Whether you are analyzing perimeter threat issues, investigating insider crimes, or are in charge of compliance monitoring and reporting, this book is meant for you. The reader should have a basic understanding of programming to follow the Perl and UNIX scripts in this book. I assume that you are familiar with basic networking concepts and have seen a log file before. You don't have to be an expert in IT security or compliance. It helps to have an understanding of the basic concepts, but it is definitely not a prerequisite for this book. Most of all, I want you to read this book with an open mind. Try to see how visualization can help you in your daily job. Structure and Content This book follows a simple organization. It introduces basic visualization and data graphing concepts first. It then integrates those concepts with security data and shows how you can apply them to security problems. In the following list, I briefly describe each chapter: Chapter 1: Visualization Visualization is the core topic of this book. The first chapter introduces some basic visualization concepts and graph design principles that help generate visually effective graphs. Chapter 2: Data Sources Visualization cannot exist without data. This chapter discusses a variety of data sources relevant to computer security. I show what type of data the various devices generate, show how to parse the data, and then discuss some of the problems associated with each of the data sources. Chapter 3: Visually Representing Data Data can be visualized in many different ways. This chapter takes a closer look at various forms of visualizations. It first discusses generic graph properties and how they can help encode information. It then delves into a discussion of specific visualizations, such as charts, box plots, parallel coordinates, links graphs, and treemaps. The chapter ends with a discussion of how to choose the right graph for the data visualization problem at hand. Chapter 4: From Data to Graphs This chapter introduces the information visualization process. It is a step-by-step process that guides you through how to take the data and generate a graphical representation of it. It also discusses how to interpret the resulting visual representation. In addition, the chapter discusses ways to process data with various tools, such as UNIX scripts or Perl. Chapter 5: Visual Security Analysis Visually analyzing security data can be separated into three classes: reporting, historical analysis, and real-time monitoring. Historical analysis I discuss in four sections: time-series visualization, correlation graphs, interactive analysis, and forensic analysis. These are the topics discussed in this chapter. Chapter 6: Perimeter Threat This chapter is a collection of use-cases. It starts out with a discussion of use-cases involving traffic-flow analysis. Everything from detecting worms to isolating denial-of-service attacks and monitoring traffic-based policies is covered. The use-cases are then extended to firewall logs, where a large firewall log is analyzed first. In a second part, firewall logs are used to assess the ruleset to find potential misconfigurations or security holes. Intrusion detection signature tuning and wireless access log analysis are the next two use-cases that deal with network layer data. The remainder of the chapter looks at application layer data. Email server logs are first analyzed to find open relays and identify email-based attacks. A second part then looks at social network analysis using email transaction logs. The chapter closes with a discussion of visualizing vulnerability scan data. Chapter 7: Compliance This chapter first introduces compliance in a log analysis context. I discuss the basics of control objectives and policies and show which federal or industry regulations require companies to analyze and collect their logs. I then show how visualization can help analyze audit data for compliance. Going through this process, it becomes necessary to start mapping the log files against business processes to weigh their importance. This leads into a risk management discussion and shows how risk-centric security visualizations can be generated. The chapter finishes up with a discussion of two compliance use-cases: the visualization of separation of duties in an application context and the monitoring of databases. Chapter 8: Insider Threat Instead of looking from the outside in, insider threat focuses on monitoring inside the perimeter. This chapter first introduces the topic and discusses different aspects of it, such as who a typical insider is. The chapter then introduces a detection framework that helps assess and monitor individuals. Through the use of so-called precursors, we can then identify potential malicious insiders and find users behaving suspiciously. Visualization is a key component of the insider detection process. Chapter 9: Data Visualization Tools After a short introduction to different data formats used by visualization tools, this chapter then surveys visualization tools and libraries. The chapter then introduces about 20 tools and open source visualization libraries that you can use in your own programs. All of these tools are also available on the accompanying CD, the Data Visualization and Analysis Linux (DAVIX). Color Color is a key property of information visualization. Unfortunately, the cost of printing a book in color is quite high. This is why the images in the book are printed in black and white. However, because color is an important graph property, the book contains an insert of 16 color pages in the middle of the book. This insert is a collection of figures from throughout the book that illustrate how color enhances the readability of the visualizations. The following table lists the figures that are featured in the color insert. Color Insert Table Figures that appear in the color insert Figure Number Page Number Figure 3-1 68 Figure 3-17 86 Figure 3-27 95 Figure 3-39 116 Figure 4-10 141 Figure 4-11 143 Figure 4-12 146 Figure 4-15 150 Figure 6-7 251 Figure 6-12 260 Figure 6-13 261 Figure 6-16 263 Figure 6-17 264 Figure 6-18 265 Figure 6-19 267 Figure 6-24 276 Figure 6-26 284 Figure 6-27 285 Figure 6-38 305 Figure 6-41 308 Figure 6-43 311 Figure 6-44 312 Figure 7-6 342 Figure 8-6 386 Figure 8-16 412 Figure 8-17 413 Figure 8-19 420 Figure 8-23 428 Figure 8-24 430 (c) Copyright Pearson Education. All rights reserved. Excerpted from Applied Security Visualization by Raffael Marty All rights reserved by the original copyright owners. Excerpts are provided for display purposes only and may not be reproduced, reprinted or distributed without the written permission of the publisher.